{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Diabetes-- Logistic 回归"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 任务描述"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "任务目的： 在 Pima Indians Diabetes Data Set（皮马印第安人糖尿病数据集）进行分类器练习"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pregnancies： 怀孕次数  \n",
    "Glucose： 口服葡萄糖耐受试验中，2 小时的血浆葡萄糖浓度。  \n",
    "BloodPressure： 舒张压（mm Hg）  \n",
    "SkinThickness： 三头肌皮肤褶层厚度（mm）  \n",
    "Insulin：2 小时血清胰岛素含量（μU/ ml）  \n",
    "BMI： 体重指数（体重，kg /（身高，m）^   \n",
    "DiabetesPedigreeFunction： 糖尿病家族史  \n",
    "Age： 年龄（岁）  \n",
    "Outcome： 输出变了/类别标签（0 或 1，出现糖尿病为 1, 否则为 0） "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "需要提交代码文件，并给出必要的结果解释。  \n",
    "1.\t对数据做数据探索分析（可参考0_EDA_ diabetes.ipynb，不计分）\n",
    "2.\t适当的特征工程（可参考1_FE_ diabetes.ipynb，不计分）\n",
    "3.\t采用5折交叉验证，分别用log似然损失和正确率，对Logistic回归模型的正则超参数调优。（各50分）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 导入工具包"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from sklearn.feature_extraction.text import  CountVectorizer\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "from scipy import sparse\n",
    "\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "from sklearn.cluster import KMeans\n",
    "from nltk.metrics import distance as distance\n",
    "\n",
    "from sklearn import model_selection,preprocessing,ensemble\n",
    "from sklearn.metrics import log_loss\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer,CountVectorizer\n",
    "\n",
    "#from sklearn.cross_validation import StratifiedKFold\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "\n",
    "#from MeanEncoder import MeanEncoder\n",
    "\n",
    "#可视化\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 读取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "#import data\n",
    "dpath = \"./\"\n",
    "data = pd.read_csv(dpath + \"FE_pima-indians-diabetes.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 准备数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = X_train[\"Target\"]\n",
    "X_train = X_train.drop([\"Target\"], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 建模 + 训练"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### default  Logistic Regression 模型训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "lr= LogisticRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "logloss of each fold is:  [0.48797856 0.53011593 0.4562292  0.422546   0.48392885]\n",
      "cv logloss is: 0.47615970944434044\n"
     ]
    }
   ],
   "source": [
    "from sklearn.cross_validation import cross_val_score\n",
    "loss = cross_val_score(lr, X_train, y_train, cv=5, scoring='neg_log_loss')\n",
    "print('logloss of each fold is: ',-loss)\n",
    "print('cv logloss is:', -loss.mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 正则化的 Logistic Regression及参数调优（GridSearchCV）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "logistic回归的需要调整超参数有：C（正则系数，一般在log域（取log后的值）均匀设置候选参数）和正则函数penalty（L2/L1） \n",
    "目标函数为：J = sum(logloss(f(xi), yi)) + C* penalty \n",
    "\n",
    "在sklearn框架下，不同学习器的参数调整步骤相同：\n",
    "设置候选参数集合\n",
    "调用GridSearchCV\n",
    "调用fit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5, error_score='raise',\n",
       "       estimator=LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False),\n",
       "       fit_params=None, iid=True, n_jobs=1,\n",
       "       param_grid={'penalty': ['l1', 'l2'], 'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
       "       scoring='neg_log_loss', verbose=0)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "#需要调优的参数\n",
    "# 请尝试将L1正则和L2正则分开，并配合合适的优化求解算法（slover）\n",
    "#tuned_parameters = {'penalty':['l1','l2'],\n",
    "#                   'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000]\n",
    "#                   }\n",
    "penaltys = ['l1','l2']\n",
    "Cs = [0.001, 0.01, 0.1, 1, 10, 100, 1000]\n",
    "tuned_parameters = dict(penalty = penaltys, C = Cs)\n",
    "\n",
    "lr_penalty= LogisticRegression()\n",
    "grid= GridSearchCV(lr_penalty, tuned_parameters,cv=5, scoring='neg_log_loss')\n",
    "grid.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'mean_fit_time': array([0.00279794, 0.0023983 , 0.00139818, 0.00179892, 0.00179915,\n",
       "        0.00200062, 0.00199909, 0.00159874, 0.00200253, 0.00179815,\n",
       "        0.00200443, 0.00200338, 0.00219512, 0.0018012 ]),\n",
       " 'std_fit_time': array([7.48226419e-04, 1.35565773e-03, 4.89317495e-04, 4.00329328e-04,\n",
       "        4.00210947e-04, 1.91567511e-06, 6.31658003e-04, 4.87579357e-04,\n",
       "        6.32447413e-04, 3.99235823e-04, 5.43761646e-06, 6.04361706e-06,\n",
       "        3.95566903e-04, 4.01019721e-04]),\n",
       " 'mean_score_time': array([0.00139952, 0.00079947, 0.00059981, 0.00079999, 0.00079999,\n",
       "        0.00079536, 0.00080256, 0.0007937 , 0.00079632, 0.00060158,\n",
       "        0.00079451, 0.00099759, 0.00080147, 0.00079389]),\n",
       " 'std_score_time': array([8.00490470e-04, 1.16545164e-03, 4.89745640e-04, 3.99995031e-04,\n",
       "        3.99997191e-04, 3.97699485e-04, 4.01334248e-04, 3.96883801e-04,\n",
       "        3.98208621e-04, 4.91216765e-04, 3.97275945e-04, 2.09699950e-06,\n",
       "        4.00762912e-04, 3.96992161e-04]),\n",
       " 'param_C': masked_array(data=[0.001, 0.001, 0.01, 0.01, 0.1, 0.1, 1, 1, 10, 10, 100,\n",
       "                    100, 1000, 1000],\n",
       "              mask=[False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'param_penalty': masked_array(data=['l1', 'l2', 'l1', 'l2', 'l1', 'l2', 'l1', 'l2', 'l1',\n",
       "                    'l2', 'l1', 'l2', 'l1', 'l2'],\n",
       "              mask=[False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'params': [{'C': 0.001, 'penalty': 'l1'},\n",
       "  {'C': 0.001, 'penalty': 'l2'},\n",
       "  {'C': 0.01, 'penalty': 'l1'},\n",
       "  {'C': 0.01, 'penalty': 'l2'},\n",
       "  {'C': 0.1, 'penalty': 'l1'},\n",
       "  {'C': 0.1, 'penalty': 'l2'},\n",
       "  {'C': 1, 'penalty': 'l1'},\n",
       "  {'C': 1, 'penalty': 'l2'},\n",
       "  {'C': 10, 'penalty': 'l1'},\n",
       "  {'C': 10, 'penalty': 'l2'},\n",
       "  {'C': 100, 'penalty': 'l1'},\n",
       "  {'C': 100, 'penalty': 'l2'},\n",
       "  {'C': 1000, 'penalty': 'l1'},\n",
       "  {'C': 1000, 'penalty': 'l2'}],\n",
       " 'split0_test_score': array([-0.69314718, -0.62523328, -0.6359066 , -0.51207677, -0.48922096,\n",
       "        -0.48350008, -0.48814887, -0.48797856, -0.48903175, -0.48902949,\n",
       "        -0.48913813, -0.4891444 , -0.48915011, -0.489156  ]),\n",
       " 'split1_test_score': array([-0.69314718, -0.6338686 , -0.64460736, -0.54322757, -0.52488138,\n",
       "        -0.52514943, -0.52916757, -0.53011593, -0.53103427, -0.53113632,\n",
       "        -0.53123496, -0.53124692, -0.53125424, -0.53125807]),\n",
       " 'split2_test_score': array([-0.69314718, -0.62566121, -0.63328109, -0.50673804, -0.45825135,\n",
       "        -0.46048749, -0.45561415, -0.4562292 , -0.4558907 , -0.45595951,\n",
       "        -0.45592651, -0.45593526, -0.45593324, -0.45593286]),\n",
       " 'split3_test_score': array([-0.69314718, -0.62087342, -0.63024185, -0.48731256, -0.43335884,\n",
       "        -0.42929827, -0.42237309, -0.422546  , -0.42195626, -0.42197953,\n",
       "        -0.42192112, -0.42192491, -0.42191722, -0.42191947]),\n",
       " 'split4_test_score': array([-0.69314718, -0.63268716, -0.63580105, -0.52538884, -0.48687031,\n",
       "        -0.4853734 , -0.48452424, -0.48392885, -0.48410837, -0.48408647,\n",
       "        -0.48411332, -0.48410727, -0.48410858, -0.48410941]),\n",
       " 'mean_test_score': array([-0.69314718, -0.62766704, -0.63597526, -0.51497115, -0.47856449,\n",
       "        -0.47681232, -0.47602422, -0.4762194 , -0.47646513, -0.47649922,\n",
       "        -0.47652788, -0.47653284, -0.47653377, -0.47653626]),\n",
       " 'std_test_score': array([1.11022302e-16, 4.89097299e-03, 4.79089060e-03, 1.86921266e-02,\n",
       "        3.09279175e-02, 3.15479636e-02, 3.55971469e-02, 3.57224624e-02,\n",
       "        3.62934923e-02, 3.63084245e-02, 3.63680556e-02, 3.63697206e-02,\n",
       "        3.63749097e-02, 3.63758791e-02]),\n",
       " 'rank_test_score': array([14, 12, 13, 11, 10,  9,  1,  2,  3,  4,  5,  6,  7,  8]),\n",
       " 'split0_train_score': array([-0.69314718, -0.62745539, -0.63279271, -0.50890565, -0.46651597,\n",
       "        -0.46237618, -0.45927716, -0.45918946, -0.45913773, -0.45913682,\n",
       "        -0.45913626, -0.45913625, -0.45913624, -0.45913624]),\n",
       " 'split1_train_score': array([-0.69314718, -0.6232271 , -0.62102867, -0.50035915, -0.45841861,\n",
       "        -0.45417418, -0.45114237, -0.45108423, -0.45103428, -0.4510337 ,\n",
       "        -0.45103316, -0.45103316, -0.45103315, -0.45103315]),\n",
       " 'split2_train_score': array([-0.69314718, -0.62777054, -0.64122227, -0.5128013 , -0.47459919,\n",
       "        -0.46996298, -0.46735835, -0.46727381, -0.46723245, -0.46723158,\n",
       "        -0.46723114, -0.46723113, -0.46723113, -0.46723113]),\n",
       " 'split3_train_score': array([-0.69314718, -0.6300762 , -0.64391789, -0.5193933 , -0.48292707,\n",
       "        -0.47815891, -0.47566783, -0.47558312, -0.47554383, -0.47554298,\n",
       "        -0.47554256, -0.47554255, -0.47554255, -0.47554255]),\n",
       " 'split4_train_score': array([-0.69314718, -0.62408812, -0.63418802, -0.50711554, -0.4669455 ,\n",
       "        -0.46359351, -0.46084077, -0.4607505 , -0.46070616, -0.46070497,\n",
       "        -0.46070451, -0.46070449, -0.46070448, -0.46070448]),\n",
       " 'mean_train_score': array([-0.69314718, -0.62652347, -0.63462991, -0.50971499, -0.46988127,\n",
       "        -0.46565315, -0.4628573 , -0.46277622, -0.46273089, -0.46273001,\n",
       "        -0.46272953, -0.46272951, -0.46272951, -0.46272951]),\n",
       " 'std_train_score': array([0.        , 0.00252359, 0.0079786 , 0.00629506, 0.00829202,\n",
       "        0.00802142, 0.00822791, 0.0082209 , 0.00822504, 0.00822498,\n",
       "        0.00822503, 0.00822503, 0.00822503, 0.00822503])}"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid.cv_results_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.47602422237262604\n",
      "{'C': 1, 'penalty': 'l1'}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(-grid.best_score_)\n",
    "print(grid.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 当前结果，最佳超参数为1，正则项是L1正则"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Wd4VNX69/HvPTPpCQRIgNCLoNRQcgBFsSMqhCIgKFKO2D2AerCex4IehaMi/hULoogCAhaK2AAFUZQSkN6NlNCCQAikksx6XswEQghkUiY7k9wfr7nYe8/aM78tmju7rLXEGINSSil1MTarAyillCr7tFgopZQqkBYLpZRSBdJioZRSqkBaLJRSShVIi4VSSqkCabFQSilVIC0WSimlCqTFQimlVIEcVgcoKREREaZBgwZWx1BKKZ+yZs2av40xkQW182qxEJFuwJuAHZhsjBmb5/03gGvdq8FAdWNMuPu9IcB/3O+9ZIyZerHvatCgAXFxcSUZXymlyj0R2eNJO68VCxGxAxOBG4EEYLWIzDfGbMlpY4x5JFf7fwFt3ctVgeeAGMAAa9z7HvdWXqWUUhfmzXsWHYBdxph4Y0wmMBPoeZH2A4HP3Ms3AYuMMcfcBWIR0M2LWZVSSl2EN4tFbWBfrvUE97bziEh9oCHwU2H3VUop5X3evGch+Wy70HjoA4AvjDHZhdlXRO4F7gWoV69eUTIqpcqB06dPk5CQQHp6utVRyqzAwEDq1KmDn59fkfb3ZrFIAOrmWq8DHLhA2wHAQ3n2vSbPvkvz7mSMmQRMAoiJidGJOZSqoBISEggLC6NBgwaI5Pe7ZsVmjOHo0aMkJCTQsGHDIn2GNy9DrQaaiEhDEfHHVRDm520kIpcCVYDfc23+AegqIlVEpArQ1b1NKaXOk56eTrVq1bRQXICIUK1atWKdeXntzMIYkyUiD+P6IW8HPjLGbBaRMUCcMSancAwEZppcU/YZY46JyIu4Cg7AGGPMMW9lVUr5Pi0UF1fcfz9e7WdhjPkW+DbPtmfzrD9/gX0/Aj7yWji37KwsVk8eQb1uI6nV4FJvf51Sqoy4/X3XxYxZ911ucRLfUOGH+zjw12aaH5qL38c3sXPDCqvjKKV8VGho6Jnlbt26ER4eTvfu3fNt+9BDD9GmTRuaN29OUFAQbdq0oU2bNnzxxReF+s61a9fy/fffFyu3pyp8sajbJJpj/edhEGp82Zs/fvna6khKKR83evRoPv300wu+P3HiRNatW8e3335L48aNWbduHevWraNv376F+h4tFqWsQfN/IMMXkWSvRvPFw1g+/0OrIymlfNj1119PWFhYkfbduXMnN910E+3bt6dLly7s2LEDgJkzZ9KyZUuio6O59tprSUtLY8yYMUyfPr1IZyWFVW4GEiyuyDqXcGrEEva+E8vlax5jcdJhrr/rKb1pppSPeeHrzWw5kFxguy0HXW1y7l1cTPNalXiuR4tiZ/PEvffey+TJk2ncuDHLly/n4YcfZuHChbzwwgssXbqUGjVqkJSURFBQEM8++yybNm1iwoQJXs+lxSKX0PBIGjy6iK0T+3FD/DgWTTzINfdPwM9htzqaUqoCSEpKYsWKFdx2221ntmVlZQHQuXNnBg8eTL9+/ejTp0+pZ9NikYdfYCjNR81n4/t3c2PiJyx54wgxD00hLDjI6mhKKQ94egZQFp+GMsYQERHBunXrznvvgw8+YOXKlSxYsIDo6Gg2bNhQqtn0nkU+xO5HqwemsvmSe7k25Ts2vdGTw0e1m4dSyruqVKlCVFQUc+bMAcDpdLJ+/XoA4uPj6dSpEy+++CJVqlRh//79hIWFcfLkyVLJpsXiQkRoMehVdsQ8T8fMVRx+uxu79uy1OpVSygdcddVV9OvXjx9//JE6derwww+eD0Axc+ZM3nvvPaKjo2nRogULFiwA4JFHHqFVq1a0atWKG264gZYtW3Ldddexfv162rZt6/Ub3JKr47RPi4mJMd6a/Gjvr59Rc/HD7KMGSX1m0b51K698j1KqaLZu3UqzZs0KtU9ZvAzlbfn9exKRNcaYmIL21XsWHqh35UASK0VS86u7CPkyliUnpnDtVddYHUspVQwVqUiUBL0M5aHqrW/AOfQ7/O3QbvFA5s37nPJyVqaUUgXRYlEIYQ3aEPzgEtL8q9Ft7QN89sm7ZDu1YCilyj8tFoUUGNGA6iOX8ndoU26Pf5oZ7zxPWmZ2wTsqpZQP02JRBLbQCGqPXMSByM7c9fcE5k14mKMndYYupVT5pcWiqPxDqPvAXBLq92FA6gyWv3kXuxMLHmJAKVVGTLnV9VIe0WJRHHY/6gz9iIOtHyQ2ayHx79zGuviDVqdSSlmgtIconzNnDq+++mqxc3tKH50tLhGi+rzC35WjuOaXZ1k7tSc/9fqU69rqREpKVVSjR48mNTWV999/P9/3J06cCMDu3bvp3r17vsN7gGtcKIcj/x/TvXv3LpmwHtIzixIScf0ITnWfRLT8Se05ffhyyUqrIymlLFKcIcqvvPJKnnnmGbp06cLbb7/NvHnz6NixI23btqVr164kJiYCMHnyZEaNGgXAoEGDGDlyJFdccQWNGjU6M1xISdIzixJUKaY/6ZUiqPfZnYQuHcgHx97m7t43Y7PpMOdKlZrvnoRDGwtud8g9EJ8n9y1qtoKbxxYvVyEkJyezbNkyAI4fP05sbCwiwnvvvcfrr7/OuHHjztsnMTGR5cuXs3HjRvr371/iZx56ZlHCApteh9/w76nkB/02DOfNj6eRkaWP1iqlPDdgwIAzy3v37qVr1660atWK8ePHs3nz5nz36dWrFyJC69at2b9/f4ln0jMLL3DUjib0wR858UEsD+x5hP975wj33vMvKgf5WR1NqfLP0zOAnDOKYd94L0sRhYSEnFl+6KGHePrpp7nllltYvHgxY8fmf3wBAQFnlr0xuoSeWXiJVG1I+MNLSK1yKY8eHcOH//c8+5PSrI6llPIxJ06coHbt2hhjmDp1qmU5tFh4U0gEVR/4geRaV/Jo2tt8/dYjbN6fZHUqpZSXFWeI8ryef/55evfuzdVXX02NGjVKMGXh6BDlpSH7NCdm3UflHV/ymelKnTve4qpLa1qdSqlyoyhDlJfly1DeUpwhyvXMojTY/ag8YDKn2j/IQFlIyvS7+GrVn1anUqpiG/ZNhSoUxaXForTYbIT2eIX0616km20VtRbcyXs/rNVhzpVSPkGLRSkL7DKCrF4fEGPbydXLB/PyrCWcznZaHUsppS5Ki4UFHG36Yx/0OY0dRxmy9R6e/fArUjKyrI6llFIXpMXCInLJdfgP/5ZqAU5G7x/JsxOnkKjDnCulyigtFlaq1Zag+38kMKwqL514mlff+j92JZ6yOpVSFcKw74cx7PthVsfwGV4tFiLSTUS2i8guEXnyAm36i8gWEdksIjNybc8WkXXu13xv5rRU1UYE3/8jJqIpr2S+wifv/JfVu49ZnUopVUg5Q5SvW7eOyy+/nBYtWtC6dWtmzZp1XtuSGKIcYO3atXz//fclkr8gXhvuQ0TswETgRiABWC0i840xW3K1aQI8BXQ2xhwXkeq5PiLNGNPGW/nKlNDqBN/7PWnT7mDMvnd4/cNjJPZ9jluja1mdTClVSMHBwXzyySc0adKEAwcO0L59e2666SbCw8PPtPF0iPKCrF27lk2bNtGtW7cSyX4x3jyz6ADsMsbEG2MygZlAzzxt7gEmGmOOAxhjEr2Yp2wLCCNoyJdkNuvDY/aZHPl8FJOX7bI6lVKqkJo2bUqTJk0AqFWrFtWrV+fIkSMe779z505uuukm2rdvT5cuXdixYwcAM2fOpGXLlkRHR3PttdeSlpbGmDFjmD59epHOSgrLmwMJ1gb25VpPADrmadMUQESWA3bgeWNMzjlVoIjEAVnAWGPMXC9mLRsc/vj3+5Cs72swdNW7LFh0gpeOj+WpHm2w6zDnSnlk3KpxbDu2rcB2OW08uW9xWdXLeKLDE4XOsmrVKjIzM2ncuLHH+9x7771MnjyZxo0bs3z5ch5++GEWLlzICy+8wNKlS6lRowZJSUkEBQXx7LPPsmnTJiZMmFDobIXlzWKR30+3vD3QHEAT4BqgDvCLiLQ0xiQB9YwxB0SkEfCTiGw0xpzT7VlE7gXuBahXr15J57eGzYbjlrE4K9Wi++L/R9W4B/l30lheueNKAv3sVqdTSnno4MGD3HXXXUydOhWbzbOLOElJSaxYsYLbbrvtzLasLNdj9Z07d2bw4MH069ePPn36eCXzxXizWCQAdXOt1wEO5NNmhTHmNPCXiGzHVTxWG2MOABhj4kVkKdAWOKdYGGMmAZPANTaUNw7CKrYrR0BYdTrNfYjwP//FQ++/zGvDulIlxN/qaEqVaZ6eAeScUUzpNqXEMyQnJ3Prrbfy0ksv0alTJ4/3M8YQERGR7z2MDz74gJUrV7JgwQKio6PZsGFDSUYukDfvWawGmohIQxHxBwYAeZ9qmgtcCyAiEbguS8WLSBURCci1vTOwhYomegC2O2bR1O8Izx8ZxciJn7PvWKrVqZRSF5GZmUnv3r3PnAUURpUqVYiKijozLarT6WT9+vUAxMfH06lTJ1588UWqVKnC/v37CQsL4+TJkyV+DPnxWrEwxmQBDwM/AFuB2caYzSIyRkRi3c1+AI6KyBZgCTDaGHMUaAbEich69/axuZ+iqlCa3IDjnwuoGZjNhNQneGbiVDYk6DDnSpVVs2fPZtmyZXz88cdnHoktzNNOM2fO5L333iM6OpoWLVqwYMECAB555BFatWpFq1atuOGGG2jZsiXXXXcd69evp23btl6/wa1DlPuKo39yemovspIPMyL7UQbeOYzrLrNubHulypKiDFHuzctQZZUOUV4RVGuM3z2LcUQ24V37/1jw6RvMWLnX6lRK+awp3aZUqEJRXFosfElYDfzu/g6pfwXj/d5l9/yXee37bWeHOZ9y69kJXZRSqgRpsfA1gZWw3/Ulzua9edrvM8J/fZ5/z/qDzCwnmw+eYPPBE1YnVEqVQ1osfJEjAFvfjzAd7mO44zu6bH6G4VOWk2ICrE6mlCqntFj4KpsNuXkcXP8cPe2/ce++J3kupS9/O8OsTqaUKoe0WPgyEbjqUej5Dp3tW3nN8S4vp/QkKTXT6mRKlXl77hrMnrsGWx3DZ2ixKA/a3okMnMmltgTe9xvP3BUFj4ujlCpZpT1E+Zw5c3j11VdLLH9BvDnchypNTbuS4KhHg6y/OL7iU5zX/A+bDj6oVKkrySHKs7KycDjy/zHdu3fvkg9/EXpmUY6kSAh/E86N6Qv5ddffVsdRqkIq7hDlV155Jc888wxdunTh7bffZt68eXTs2JG2bdvStWtXEhNdMzlMnjyZUaNGATBo0CBGjhzJFVdcQaNGjc4MF1KS9MyCctSTU4R0Wwgt2c1LPy+iS9M7rE6kVKk79PLLZGwt+FJs+jZXG0/uWwQ0u4yaTz9d6CxFGaIcXAMRLlu2DIDjx48TGxuLiPDee+/x+uuvM27cuPP2SUxMZPny5WzcuJH+/fuX+JmHFotypEVUZXCGcHr/3zTa+wUHknpTKzzI6lhKVUhFGaI8x4ABA84s7927l/79+3Po0CEyMjJo2rRpvvv06tULEaF169bs37+/WNnzU+GLhdM4aTJvPZcetoP3Zyb0PpuDzKax9Ni6gCm/b2PEzW2tTqRUqfL0DCDnjKL+p5+UeIaiDlGeIyQk5MzyQw89xNNPP80tt9zC4sWLGTt2bL77BASc7WfljTH/Kvw9i71HdpHmzOC3huXncdOQy+8mTNJIiptNZpbT6jhKVSjFGaI8PydOnKB27doYY5g6dWoJJCyaCl8sameF0v8XJ013n2b7se1WxymeYd+4XvU6kVKpMd1PL2ThlkNWp1KqQinuEOV5Pf/88/Tu3Zurr76aGjWsG2lahygHfuncmoijp/n03635790zEfH9R06dy9/Ctug//DvyPV57aKDVcZTyqqIMUe7Ny1BllQ5RXkzTRkdzKsRGu9kb+GnPj1bHKRG2NneQLQ5aHJzDzsOlM5OWUr6k/qefVKhCUVxaLID3e31Kw0efosVe+G7ai2RkZ1gdqfhCqpHV9FZ6239l5m87rE6jlPJxWizcqvW/ney6Nen2bSLTNnxsdZwSEdBhGOGSwql1c0nJyLI6jlJeVV4uqXtLcf/9aLFwEz8/6j/1LLWPwa5P3uNIquc9LsushleTEVqXXs7FzFt3wOo0SnlNYGAgR48e1YJxAcYYjh49SmBgYJE/o8L3s8gt9NprsLVvTa+fN/DOr6/xXNfze0n6FJsN/w5DufynF/lw+W8M7FC3XNy8VyqvOnXqkJCQUKhhNSqawMBA6tSpU+T9tVjkIiLUe+pZnH374vfZ12xqdxctI1paHatYpO0gnEteJubYAtbuvYn29atYHUmpEufn50fDhg2tjlGu6WWoPIJatiC4+y10X21494cxvn9aG1YT5yVd6Wdfxszfd1mdRinlo7RY5KPWo49htzmI/moT3/31ndVxis3xj2FUkxOkbfqGYynlp6e6Uqr0aLHIh1+tWkQMGcpVmw2fzx9L6ulUqyMVzyU3cDokin7yI7Pj9lmdRinlg7RYXEDEffdhwitx67dH+HiTjw9dbrPjFzOYq+wbWfz7apxOH7+0ppQqdVosLsAeGkrUyFG02At/zPmAg6cOWh2peNoOQoCrTv3Azzv1iRGlVOFosbiI8L59sTWox8DFmUxY9brVcYonvB6m0XXc7viZGb/HW51GKeVjtFhchPj5UevxJ4k6Zsie+x1rD6+1OlKx2GKGUJOjZO9czL5jPn4fRilVqrRYFCD02msI6BBD/1/hjZ//i9P48PwQTW8mOziCAbaf+GzVXqvTKKV8iBaLAogIUU88SViqk+bfbmXernlWRyo6hz/2tndyvf0Pfly9gYysbKsTKaV8hFeLhYh0E5HtIrJLRJ68QJv+IrJFRDaLyIxc24eIyE73a4g3cxYkqEULKsXG0j0OPv1pPKcyT1kZp3jaDcGOk+vTF/H9Jp0YSSnlGa8VCxGxAxOBm4HmwEARaZ6nTRPgKaCzMaYFMMq9vSrwHNAR6AA8JyKWjlNR/ZFR2G0Oui08yqSNk6yMUjzVGmPqX8kg/5+Z8ftui8MopXyFN88sOgC7jDHxxphMYCbQM0+be4CJxpjjAMaYRPf2m4BFxphj7vcWAd28mLVAflFRRAz7J1dtNixf/An7kn23c5u0H0otcxj7vl/ZdijZ6jhKKR/gzWJRG8j9EzXBvS23pkBTEVkuIitEpFsh9i111e4ZjlQJZ9CPWby2+lWr4xRdsx44A8O507GE6Sv0RrdSqmAeFQsR6SwiIe7lQSIyXkTqF7RbPtvydh12AE2Aa4CBwGQRCfdwX0TkXhGJE5G40hia2B4aSo0RI7hsbzbJP/3IioMrvP6dXuEXiC16ADfZ4/hp7RZO6cRISqkCeHpm8S6QKiLRwOPAHqCgyWsTgLq51usAeWfgSQDmGWNOG2P+ArbjKh6e7IsxZpIxJsYYExMZGenhoRRPeL9++DVqyNCfbbz6+1iynD76g7bdEBzmNN2ylzLnj/1Wp1FKlXGeFoss4xqruyfwpjHmTSCsgH1WA01EpKGI+AMDgPl52swFrgUQkQhcl6XigR+AriJSxX1ju6t7m+XE4aDG6NFU/zuLhkt28sWOL6yOVDQ1mmPq/IOhAT8z/ffdvj8Uu1LKqzwtFidF5ClgEPCN+0knv4vtYIzJAh7G9UN+KzDbGLNZRMaISKy72Q/AURHZAiwBRhtjjhpjjgEv4io4q4Ex7m1lQug11xDcsSMDf7Px0Yq3OJFxwupIRSLthlDXmUBoYhxxe45bHUcpVYaJJ79RikhN4A5gtTHmFxGpB1xjjCnoUlSpiYmJMXFxcaX2fWmbN7O7bz/mdRJ44C6e7JBvN5KyLTMF81pT5me246fLxvDmgLZWJ1JKlTIRWWOMiSmoncdnFrguP/0iIk2BNsBnxQno64JatKBybCzdV8PiFZ/xZ9KfVkcqPP8QpFU/bpGV/LJxF3+fyrA6kVKqjPK0WCwDAkSkNvAjMAz42FuhfEXkqJHY7Q7uXAavrn7VN6/7tx+Cn8ngFpYza7Xv9h1RSnmXp8VCjDGpQB/gLWNMb6CF92L5Br+oKKoNHUanTZkcivuVX/b/YnWkwqvVFmq2ZnjQMmas2EO2ToyklMqHx8VCRC4H7gS+cW+zeyeSb6l2z3Ds1apyzzI//rdqHKezT1sdqfDaDaZB1p9UTd7C0u2JBbdXSlU4nhaLUbjGcJrjfqKpEa6nlyo8e2gokf/6F43+Sqf6mt3M2Daj4J3Kmtb9MY4ghgX9zLQVe6xOo5QqgzwqFsaYn40xscA7IhLqHu9phJez+Yzwvn3xb9SI4csC+GDtuxxNO2p1pMIJrIy06E13lrNqxz6dGEkpdR5Ph/toJSJ/AJuALSKyRkQq/D2LHOJwUH30v6lyJI3Oq1N4e93bVkcqvPZD8Hem0sO+gukrdbwopdS5PL0M9T7wqDGmvjGmHvAY8IH3Yvme0GuuIbhTJwb+Zue7DV+w7dg2qyMVTt2OEHEp94b8wuy4faSf1omRlFJneVosQowxZ+5RGGOWAiFeSeSjRIQaj48mICWT21f5MW7VON96lFYE2g+hUcZWIlN38d2mg1YnUkqVIZ4Wi3gR+X8i0sD9+g/wlzeD+aLA5s2pHBtL11Wn2bN9NYv2LLI6UuG0HoCx+3Nv6K9M06HLlVK5eFos/glEAl8Bc9zLw7wVypdFjhqJ3Wbnnt9DGL9mPOlZ6VZH8lxINaRZD241y9i05zBbDujESEopF0+fhjpujBlhjGlnjGlrjBmZM7udOpdfVBRVhw2lzbpkAnck8MmWMjN8lmfaDSEwK5kefnFMW6mP0SqlXC5aLETkaxGZf6FXaYX0NdWG34O9WjVGLK/E5A0fkJjqQx3dGlwFVRrwQNivzP1jP8npPtjJUClV4go6s3gNeP0iL5UPe2gIkf96mKhdx4nelsmba9+0OpLnbDZoN5jGqeuocTqBOWt1YiSlVAHFwt0Z74Kv0grpi8L79sW/cWPu+zWIb3bMY8ORDVZH8lybO0Hs/Cv8N6at2ONbT3UppbzC0055G0VkQ57XLyLyhohU83ZIX5TTUS/k0Al6bwpm3KpxOI3T6lieCasJl97MLdlL2J2YxMq/ysy8U0opi3j6NNR3uAYQvNP9+hr4BTiEDlV+QaFXX01wp070+TWbXQnr+Sb+m4J3KivaDSEw8xixget0vCillMfForMx5iljzEb36xngamPMOKCB9+L5tpyOeo6TadzzR1UmrJlA6mkfGXfpkuuhUh0eqLSc7zcdIvGkDz0CrJQqcZ4Wi1AR6ZizIiIdgFD3alaJpypHAps3p3LPnlyx/Djm4GE+3PSh1ZE8Y7ND20E0Tl5FTZPIbJ0YSakKzdNiMRyYLCJ/ichuYDIwXERCgFe8Fa68iBw1EpvNzmNrajJ181T2n/KRJ4zaDkKAxyJWMWPlXrKyfeSei1KqxHnaKW+1MaYVrrm32xhjWru3pRhjZns3ou/zq1mTqsOG0mjVfhrvdzI+brzVkTwTXhcuuYGbsxZz+EQKP23zof4iSqkS5enTUJVFZDyu+bcXi8jrIlLZu9HKl5yOeqN+q8LC3T8QdyjO6kieaTeYwLTD9ArdyjQdulypCsvTy1AfASeB/u5XMjDFW6HKI1dHvX8Rvv0AXfeGM271OLKdPjAM+KU3Q0h1Hqz0K8t2HGH33ylWJ1JKWcDTYtHYGPOce4a8eGPMC0AjbwYrj8L73oZ/48YMWSrsPLKVubvmWh2pYHY/aHMHjY4vJ8p2nBmr9OxCqYrI02KRJiJX5qyISGcgzTuRyq+cjnp+B/5m2M7a/N8f/8fJzJNWxypYu8GIyeapqDU6MZJSFZSnxeIBYKKI7BaRPcDbwP3ei1V+hV59NcGXd+LGn46TkXSMSRsmWR2pYNUaQ4Or6JqxiBOpGSzYoBMjKVXRePo01DpjTDTQGmjlHqZ8vXejlU+ujnqPI8mneHxrY6ZtncaeZB/oId1+KIGn9tG3yp/ao1upCshxsTdF5NELbAfAGOMjz4CWLYHNmlG5Z0+affMNtZoG8trq13jr+resjnVxl3WHoCo8WGk51+5pwqb9J2hZWx+IU6qiKOjMIqyAlyqiyFEjEbudJ9fUZmnCUn7b/5vVkS7OLxCiB9LgyBJq+aXo2YVSFcxFzyzcTz0pL8jpqGfefY/OraP43+r/8UXUFzhsF/0rsVa7wciKd3im9joeWxfGU7c0o3KQn9WplFKlwNMb3GeIyNpCtO0mIttFZJeIPJnP+0NF5IiIrHO/hud6LzvX9nI5K1+1u4djr1aNB34J5s+kXczeXsY7w1dvBnU6cEPa96SfzuartQlWJ1JKlZJCFwtAPGokYgcmAjcDzYGBItI8n6azjDFt3K/Juban5doeW4ScZV5ORz3/TX9y55EmTFw3kaT0JKtjXVz7IQSc+JM7auznU50YSakKoyjFwtNJGToAu9yd+DKBmUDPInxfuRbe9zb8L2lMzx9OkJZ+knfWv2N1pItr0RsCKnFf2K/EH0nh9z+PWp1IKVUKCl0sjDH/8bBpbSD3uNYJ7m153eaeee8LEamba3ugiMSJyAoR6VXYnL5CHA5qjB4NCQcZndCa2dtns+v4LqtjXZh/CLTqS71DC6kblMG0lXqjW6mKwNOBBE+KSHKe1z4RmSMiFxr2I7/LVXmvWXwNNDDGtAYWA1NzvVfPGBMD3AFMEJHG+eS6111Q4o4cOeLJoZRJIV26EHx5J9ou2ElEdhD/W/2/sn15p90QJCud/9TbzA+bD3M4WSdGUqq88/TMYjwwGteZQR3g38AHuC4tfXSBfRKA3GcKdYADuRsYY44aYzLcqx8A7XO9d8D9ZzywFGib9wuMMZOMMTHGmJjIyEgPD6XsyemoZ5JP8p9tl/H7wd9Zum+p1bEurFYbiIrm2pRvyXY6mblKJ0ZSqrzztFh0M8a8b4w5aYxJNsZMAm4xxswCqlxgn9VAExFpKCJSkqAgAAAeCklEQVT+wADgnKeaRCQq12ossNW9vYqIBLiXI4DOwBaPj8oHBTZrRuVevajx7RraZ9fltbjXyMzOtDrWhbUbgv/fWxhS/zgzVu3htE6MpFS55mmxcIpIfxGxuV/9c72X7/USY0wW8DDwA64iMNsYs1lExohIztNNI0Rks4isB0YAQ93bmwFx7u1LgLHGmHJdLAAiR45A7HZGrY5k78m9TN863epIF9aqH/gFc0/IMg4nZ/Dj1sNWJ1JKeZF4cm3cfV/iTeByXMVhBfAIsB9ob4z51ZshPRETE2Pi4nxkQqGLSHzzTY6++x5f/DuGb4J3sqD3AiKCIqyOlb+5D2K2zOMG3ieqeiTThncseB+lVJkiImvc94cvytOBBOONMT2MMRHGmEj38i5jTFpZKBTlSbW7h2OPiKD/wlQystJ5648yPGZUuyFI5imeqb+VX3f9TfyRU1YnUkp5iadPQzUVkR9FZJN7vbWIePoIrSqEnI56zg1beOzUlczZOYctR8voFbi6HSDyMq46+Q0OmzBdp11Vqtzy9J7FB8BTwGkAY8wGXDeslReE39YH/0sa03HuDiIc4YxbNa5sPkorAu2G4HfoD/7ZJIXP4/aRlqkTIylVHnlaLIKNMavybMsq6TDKJaejXtbeffzn4D9Ym7iWH/b8YHWs/EUPALs/w4J+ITk9i683HCh4H6WUz/G0WPzt7hRnAESkL6DTpXlRSJcuhFxxObU/X050wCWMjxtPelYZ7PwWXBWaxVJz91xaRPrp0OVKlVOeFouHgPeBy0RkPzAKnVbVq0SE6qNH40xOZvTWhhxMOcjHmz+2Olb+2g9B0k/wZIMdbEg4wfp9ZXwwRKVUoXlaLPYDU4D/4uq1vQgY4q1QyiWno17AV4vpE9KZjzZ9xKGUQ1bHOl+Dq6BqIy5PWkCwv13PLpQqhzwtFvOAHrhucB8ATgEp3gqlzoocNRLsdgb9LGQ7s5mwdoLVkc4nAu0G49j3O8ObZTN//QGSUstw73OlVKF5WizqGGMGGGP+Z4x5Pefl1WQKAL8aNaj2z2FkLVrKCP+b+Cb+G9Ylrsu37bDvhzHs+2GlnNAt+g6wORgc8DMZWU6+WKMTIylVnnhaLH4TkVZeTaIuqOo/78YeEcFVc+OJDIxg3KpxOE0ZG4sprAY07UbEri/pWC+U6Sv34nSWwcd9lVJF4mmxuBJY454idYOIbBSRDd4Mps7K6aiX8cd6/pNxI5uObmJB/AKrY52v/VBI/ZvH6u3ir79TWP7n31YnUkqVEE+Lxc1AE6ArrnsX3d1/qlISflsfAppcQr1pS4kOb8GENRNIPZ1qdaxzNb4OKtcl5tjXVA3x1xvdSpUjno4NtSe/l7fDqbPE4aD66NGc3ruPJ/ZHcyTtCJM3Ti54x9Jks0PbQdjil3JPKzuLthzm4Ik0q1MppUpAUebgVhYJueoqQq64nICp8+hToytTN08l4WQZu5HcdhCIcIffzxjgM50YSalyQYuFDxERqj/+ONnJyQyNC8VuszN+zXirY52rch245AYqb5vFdU2qMnPVXp0YSalyQIuFjwm87DIq9+5Nxqw5PBTZl0V7FrH60GqrY52r3RA4eZAR9XaTeDKDRVt0YiSlfJ0WCx8UOXIE2O1c++1+aoXUYuyqsWQ7y9Bor01vgpDqtE6cS+3wID79XW9vKeXrtFj4oJyOeinfL+TJkD7sOL6DL3d+aXWss+x+0PZOZOdC7m0byO/xR9mVeNLqVEqpYtBi4aNyOuo1/ORn2ldvx9t/vE2WswyNGt9uMBgnfW0/42cXpq3QiZGU8mVaLHyUPTSEyBH/Iu2PP3gi5WqSMpI4mFKGRo2v2ggadiFk8wxubVmDL9ckkJpZhoqZUqpQtFj4sPA+ro56fpNm0bdhTxJTE8vWnBfthkDSXh6on8DJjCzmr9OJkZTyVVosfNjZjnp7GbajFjaxsffk3rIzBWuzHhBUlaYJX3FZzTA++X1P2cmmlCoULRY+ztVR7wpSP5hKr1WQnJnM0O+Hsv3YdqujgSMAogci277h7nZhbDmYzB86MZJSPkmLhY9zddRzzajXbbWTvnF+/HXiL/ov6M9LK14iKd3iH87th4DzNLH8TIhOjKSUz9JiUQ7kdNSrdOI0nbfb+Lr31wy4dABf7PiC7nO7M2vbLOv6YUReCnU7EbB+Gn3a1mbBhoMcT9GJkXxVxym30XHKbVbHKLbychxQeseixaKciBw5AiMQfjSDygGVearjU8zuMZumVZry0sqXuH3B7aw5vMaacO0Gw9Gd3FP/EJlZTj5fU7HGiypPP5hUxaXFopzwq1GDk+H+hKRkcfD553GmpNC0SlM+7Pohr139GicyTzD0+6E8vuxxDqeU8vAbLXpBQCXq7f6cDg2qMm2FToyklK/RYlGOnKjiT3JlP5JmzSa+Zy9SV69GRLipwU3M6zmP+1rfx497fqTH3B5M3jiZzOxSuhzkHwKt+sGWeQxtV5m9x1JZtvPIRXe5/f3fuf3930snn1KqQFosyhMRkiICqT/tUxBhz+AhHH5lLM70dIL9gnm47cPM7TWXy6Mu5821b9JrXi9+3vdz6WRrPwSy0rkxexkRof4F9uje7f8au/1fK51sSqkCabEoh4Lbt6fR3DlUGTiAY1On8lfvPqRtcM2CWzesLm9e9ybv3/A+drHz8E8P8+DiB9l9Yrd3Q0VFQ1Qb/NZ9yu0xdfhp22H2J+nESEr5Cq8WCxHp5p63e5eIPJnP+0NF5IiIrHO/hud6b4iI7HS/hngzZ3lkCwmh5rPPUu+jD3Gmp7N7wEAS35iAM9N16emK2lfwVexX/Dvm36xNXEvv+b15Y80bpJxO8V6o9kPg8CYG1z/mmhhppY4XpZSv8FqxEBE7MBHX/N3NgYEi0jyfprOMMW3cr8nufasCzwEdgQ7AcyJSxVtZy4uZ/2rBzH+1OGdbyBVX0Gj+PCr36sXR999nd7/+pG/bBoCf3Y8hLYawoPcCbm14Kx9t+ogec3qwIH6Bd3pat+wLfsHU2DmT6y+rzszVe8nM0omRlPIF3jyz6ADsMsbEG2MygZlATw/3vQlYZIw5Zow5DiwCunkpZ7lnDwuj1sv/pc6775B17Ch/9evP3++9h8lyDewXERTBS1e+xLRbplE9uDpP/fIUQ74fwtajW0s2SGAlaNkHNn7J4PbV+PtUJj9sPlSy36GU8gpvFovaQO4H6hPc2/K6TUQ2iMgXIlK3kPuqQgi79loazZ9PpRtv5MiEN9k98A4y/vzzzPvRkdHMuHUGL1zxAnuS93D7gtsZ8/sYjqcfL7kQ7YbC6RSuTF9G3apBfKo9upXyCd4sFpLPtrzXNr4GGhhjWgOLgamF2BcRuVdE4kQk7siRiz+KqVwcVapQe/zr1H5jPKf37eOv3n04OuVjTLarh7dNbPRp0oeve3/Nnc3u5KudX9F9Tnc+2/ZZycyXUScGIpthWzuVQR3rs+qvY2w/pBMjKVXWebNYJAB1c63XAc4Zo9oYc9QYk+Fe/QBo7+m+7v0nGWNijDExkZGRJRa8Iqh08800+no+IVdeSeK4cewZMoTMvWdvOFfyr8QTHZ7gix5f0KxqM15e+TK3L7i9+PN9i7hudB9Yy4B6J/B32Ji+Us8ulCrrvFksVgNNRKShiPgDA4D5uRuISFSu1Vgg5yL5D0BXEanivrHd1b1NlSBHZCR1Jr5N1NhXyNi+g/hevTn+2Wfn3Ny+pMolfND1A8ZfM56TmSf55w//ZPTPozmUUox7Da1vB3sAlbd+RvdWUXy1dj8pGToxklJlmdeKhTEmC3gY1w/5rcBsY8xmERkjIrHuZiNEZLOIrAdGAEPd+x4DXsRVcFYDY9zb1EVM6TaFKd2mFGofESG8Vy8azZ9HcJs2HHphDPvuHs7pgwfPaXNj/RuZ12seD0Q/wJJ9S4idG8ukDZPIyM64yKdfQHBVaB4LG2YxKKYGpzKymLtuf+E/RylVarzaz8IY860xpqkxprEx5r/ubc8aY+a7l58yxrQwxkQbY641xmzLte9HxphL3K/C/QRUheYXFUXdDydT8/nnSV23jvgesSR9Neecs4wgRxAPtnmQeb3m0blWZ9764y16ze3Fkr1LCv+obbshkH6CtieX0jyqEp/qxEhKlWnag1udISJUGXA7jebNJeCySzn49NMkPPgQWXkeHqgdWps3rn2DSTdOwt/uz4glI3jgxwf468Rfnn9ZgyuhamNk7ScM6lSfbYdOsmZPCT51pZQqUVos1Hn869al/iefUP3JJ0j57Tfiu/cg+bvvzmt3ea3L+SL2Cx7/x+OsT1xPn3l9eD3udU5lnir4S0RcQ5fv/Y1edU8RFuDQiZGUKsO0WKh8ic1GtaFDaTjnK/zq12f/I4+y/9FHyTp+7m//fjY/7mp+F1/3/poejXvw8eaP6TG3B1//+TVOU0Dv7DZ3gM1B8Mbp3Na+Dt9uPMTRU0W4B6KU8jotFuqiAho1osGM6USOGkXyosXE94jl5E8/ndcuIiiCMZ3HMOOWGUSFRPH0r08z+LvBbD66+cIfHlodLr0Z1n/GoJgaZGY7mR2X4MWjUUoVlRYLVSBxOIi4/z4afj4bR0QECQ8+xIGnnib75Pmd6VpFtmLaLdN4sfOL7Du5j4ELBvL8b89zLP0CD7O1GwqpR7nk+C90alSV6Sv3kK0TIylV5mixUB4LvOwyGs6eRbUH7ufE/PnE94jl1PLl57WziY1el/RiQe8F3NX8Lubtmkf3Od2ZvnX6+b3AG18LlevCmqkM6lSfhONpLNuhvfGVKmu0WKhCEX9/qo8cSYPPZmALDmbf3cM5+MILOFPOH9o8zD+M0f8YzZexX9KiWgvGrhpLv6/7sergqrONbHZoexfEL6FrVAaRYQE6XpRSZZAWC1UkQa1b0/CrL6k6bBhJM2cR36s3qXFx+bZtFN6ISTdOYsI1E0jLSuPuhXfz2NLHOHjK3fGv7SAQG/4bpjHwH3VZsj2R7MzQUjwapVRBtFioIrMFBlLjicep/+knAOy5azCHx/0PZ3r6eW1FhOvrX8/cnnN5qM1DLEtYRuzcWN5b/x4ZoRFwyY2wbjoDYmohQOqJS0v5aJRSF6PFQhVbcEwMjebOIXzA7RybMoW/+tx2ZhrXvAIdgdwffT/zes2jS50uTFw3kZ5ze/JjwxjMyYPUSvyFG5rVIO1EU4xT//NUqqzQ/xtVibCFhBD13HPU/XAyztRUdg+8g8QJEzDuaVzzqhVai9eveZ3JXScT5Ahi1M5Pub92HeLXTGJQp/qY7CDSTzUo3YNQSl2Qw+oAqnwJ7dyZRvPncfiVsRx9731OLf2ZWmNfIfCyy/Jt3zGqI5/3+JxZ22cxcfXr3HZ6F3ckvo3dvwEnD19O7Nu/EuhnJyjn5W8/u+5vI8jPtR7s7zhnPaftmfVcy3ZbftOlKKUuRouFKnH2SpWo9crLhN14IweffZa/+vUn8qGHqDb8bsRx/n9yDpuDO5vdyc3hzfm/L/vw6Z9fEdrQjjnZnMyQpqRkQ9ZpyEqH09nuV5YhM8tgEIwRwAZGcM2blXtZwAgGAWMDBIfdToDdToDdQYCfA3+7nQCHnUA/BwEOh2vZ4Uegn51Ah8O17O8gyOEgyM9BoJ+DID8/V4HycxDs739mOcTfjyB/BwF2Ow67Axs2jHvertPO02DA5Pxjzv4J4DTOc94DzrbJ095pnDiNazk717rTaXAaQ5bJxuk0GOP63GzjxOledu3jJNsYjNO9n/tzjHv5zOcZQ7bz7HdlZPqBET7fuMyVL9ffY87yhQaEzPn3cP7b527I+745f96zM23OfuaFvjP/z03PCAQMH61ZmO9+viQ9IwAphd9/pLyM9BkTE2PiLvA0jrJO1vHjHH7xRZK//Y7A1q2pNfYVAho1uvAOU2PZfOIvhoVBml4kVcojzvSabL5vUZH2FZE1xpiYgtrpmYXyKtc0ruMJu+EGDr0whr969yHykVFUHTwYseVTDdoPocUX/ySWGqwLDOLTf65w/yacneu34mwMOb/95vPCidPp+vNMe/dvyzn7ZjvPfkbu98/5DPdyttNJRlYW6VlZZGRlk5F1mows17aMrCwys7LJzM4mIzuL0+7lzOwsTme7ljcf2QUITas2QLAhAoIgImf+tInNdR4krvdtnH3P9b649xVs57Rzf16udZu7PciZ5bzfZZdcn40Nm3A2x5nvc63n/ozJf8wC4L72A8/8leX+pVZy1i7wm27O+5LPr8LnfM55b8vZz87b3t34Qr9cn/tZrpUJKz8C4NFOd19gL98xfsWHOErhzEKLhSoVlW65heB//IODzz5H4thxnFr8I1GvvIx/3brnNrysOwRV5frUk2wIDCbIEWRN4BLUccptAMy983lrg5SAadtcU8s82PEWi5MUz7vr3wFgcNvrLE5SfBPXTSyV79ETfVVqHJGR1HlnIlGvvEL6tm3E9+zF8Zkzz73e7AiANnfQMTWVe+ekWRdWKXUOLRaqVIkI4b170ejr+a5pXJ9/gX3D7zlnGlfaDcZmg9rVU60LqpQ6h97gVpYxxpA0axaH//cqYrdT45mnqdyzJyLCsUciqVo5E4IjoFKtc19hOcu1oVIUBIRZfSgX9e0NzQG4ZfEWi5MUX3k5lvJyHFD8Y9Eb3KrMc03jOoCQK67gwNNPc/DJpzi5aDFRLzzPxl3hREWk0bRPD0g+AMn7IWE1pB49/4MCKrmLSJS7gNRyFZEzy7UhqEp+d02VUh7SYqEs51+vHvWnTuXYJ59y5I03iO/eA2NzsCutEk17TDi38el0OHkAkg+eLSInD7r+TD4Af26HU4cg7yx9jsBcxSTqbBHJXWBCq7tGwVVKnUeLhSoTxG6n2rChhHa5igNPPkWNjUmknzLse+BBbEFB2EKCkaAgbEHB2IKDXduCg7AFN0WCorFVDXGtBwVhCwhAzElsWUlIWiJyppi4C8y+Va4Ck52ZNwSE1cznLCX3JbAo1014pSoYLRaqTAlo3JgGn83gtytaEZwOpw8dwqSm4kxNxZmWhjMtDbKzPf9Au91VQIKCsAUHI8HB2IJaYQvuiM3fjs3uRBxObJKBjXRsSanYnKeQ7I3YTi/DRho2h8HmcGJzGMRhsIVVxVY1Cgmvc+GzlAAdYl2VL1osVJkjDgcnKgknKkHbOV+d854xBpOZiTM1FZOWdraIpKbhTE1xbUtLw5mSU1xSz7ZNOVtwspOTyUpLde/n2m7OG1o92P3Kz1HEfhSbYx1iz3YXlFwvfwcSHIQtJJSmtjQMcGREz9xHec4fnq/nue9ykXVBCtg/13J+223nt2sk6Rjg76cH48saiuvv2tePA1zHklUK451psVA+RUSQgABsAQFQpUqJfrbJzsaZlo5JyylA7mKSu+CcKS65tp06hTP5GM5TJ3CmnOR0SgomJQ3n0UycmUk4MwPACH/v3VGiea3hugR3ZN9qi3MUV3k5DoAAgsJPe/1btFioMilDSr/nttjt2ENDIDSkRD+3sI825vs4+4UecS/M9tzbzozEZ/LdbnDm2cf158LYDgB0nbsi/+/1EQt7dQJ8/zgg51j8uMTL36PFQqkyJr9xk0r7sd8LfZvT/Y4ttHLphfGC8nIccPZYvE17cCullCqQnlmoMmnsHQ0A6G1tDKWUm55ZKKWUKpAWC6WUUgXy6mUoEekGvAnYgcnGmLEXaNcX+Bz4hzEmTkQaAFuB7e4mK4wx93szq1Le8sKdzQDw7RkgXMrLsZSX44DSOxavFQsRsQMTgRuBBGC1iMw3xmzJ0y4MGAGszPMRfxpj2ngrn1JKKc958zJUB2CXMSbeGJMJzAR65tPuReB/QN7us0oppcoIbxaL2sC+XOsJ7m1niEhboK4xZkE++zcUkT9E5GcRuSq/LxCRe0UkTkTijhw5UmLBlVJKncubxSK/niJnuoqKiA14A3gsn3YHgXrGmLbAo8AMEal03ocZM8kYE2OMiYmMjCyh2EoppfLy5g3uBKBurvU6wIFc62FAS2Cpu8dqTWC+iMQaY+KADABjzBoR+RNoCuhUeBVE86jzfjdQSlnIm2cWq4EmItJQRPyBAcD8nDeNMSeMMRHGmAbGmAbACiDW/TRUpPsGOSLSCGgCxHsxq1JKqYvw2pmFMSZLRB4GfsD16OxHxpjNIjIGiDPGzL/I7l2AMSKSBWQD9xtjjnkrqyp7pnSbYnUEpVQuXu1nYYz5Fvg2z7ZnL9D2mlzLXwJfejObUkopz2kPbqWUUgXSYqGUUqpAku9EKz4oJibGxMXpw1JKKVUYIrLGGBNTUDs9s1BKKVUgLRZKKaUKpMVCKaVUgbRYKKWUKpAWC6WUUgXSYqGUUqpAWiyUUkoVSIuFUkqpAmmxUEopVaBy04NbRI4Ae4rxERHA3yUUx0rl5ThAj6WsKi/HUl6OA4p3LPWNMQXOHlduikVxiUicJ13ey7rychygx1JWlZdjKS/HAaVzLHoZSimlVIG0WCillCqQFouzJlkdoISUl+MAPZayqrwcS3k5DiiFY9F7FkoppQqkZxZKKaUKpMXCTUReFJENIrJORBaKSC2rMxWViLwqItvcxzNHRMKtzlRUItJPRDaLiFNEfO7JFRHpJiLbRWSXiDxpdZ7iEJGPRCRRRDZZnaU4RKSuiCwRka3u/7ZGWp2pqEQkUERWich697G84LXv0stQLiJSyRiT7F4eATQ3xtxvcawiEZGuwE/GmCwRGQdgjHnC4lhFIiLNACfwPvBvY4zPTIcoInZgB3AjkACsBgYaY7ZYGqyIRKQLcAr4xBjT0uo8RSUiUUCUMWatiIQBa4Bevvj3IiIChBhjTomIH/ArMNIYs6Kkv0vPLNxyCoVbCOCzVdQYs9AYk+VeXQHUsTJPcRhjthpjtludo4g6ALuMMfHGmExgJtDT4kxFZoxZBhyzOkdxGWMOGmPWupdPAluB2tamKhrjcsq96ud+eeVnlxaLXETkvyKyD7gTeNbqPCXkn8B3VoeooGoD+3KtJ+CjP5TKKxFpALQFVlqbpOhExC4i64BEYJExxivHUqGKhYgsFpFN+bx6AhhjnjHG1AWmAw9bm/biCjoWd5tngCxcx1NmeXIsPkry2eazZ6zljYiEAl8Co/JcWfApxphsY0wbXFcQOoiIVy4ROrzxoWWVMeYGD5vOAL4BnvNinGIp6FhEZAjQHbjelPEbU4X4e/E1CUDdXOt1gAMWZVG5uK/vfwlMN8Z8ZXWekmCMSRKRpUA3oMQfQqhQZxYXIyJNcq3GAtusylJcItINeAKINcakWp2nAlsNNBGRhiLiDwwA5lucqcJz3xT+ENhqjBlvdZ7iEJHInKcdRSQIuAEv/ezSp6HcRORL4FJcT97sAe43xuy3NlXRiMguIAA46t60woef7OoNvAVEAknAOmPMTdam8pyI3AJMAOzAR8aY/1ocqchE5DPgGlwjnB4GnjPGfGhpqCIQkSuBX4CNuP5/B3jaGPOtdamKRkRaA1Nx/fdlA2YbY8Z45bu0WCillCqIXoZSSilVIC0WSimlCqTFQimlVIG0WCillCqQFgullFIF0mKhVCGIyKmCW110/y9EpJF7OVRE3heRP90jhi4TkY4i4u9erlCdZlXZpsVCqVIiIi0AuzEm3r1pMq6B+ZoYY1oAQ4EI96CDPwK3WxJUqXxosVCqCMTlVfcYVhtF5Hb3dpuIvOM+U1ggIt+KSF/3bncC89ztGgMdgf8YY5wA7tFpv3G3netur1SZoKe5ShVNH6ANEI2rR/NqEVkGdAYaAK2A6riGv/7IvU9n4DP3cgtcvdGzL/D5m4B/eCW5UkWgZxZKFc2VwGfuET8PAz/j+uF+JfC5McZpjDkELMm1TxRwxJMPdxeRTPfkPEpZTouFUkWT3/DjF9sOkAYEupc3A9EicrH/BwOA9CJkU6rEabFQqmiWAbe7J56JBLoAq3BNa3mb+95FDVwD7+XYClwCYIz5E4gDXnCPgoqINMmZw0NEqgFHjDGnS+uAlLoYLRZKFc0cYAOwHvgJeNx92elLXPNYbMI1b/hK4IR7n284t3gMB2oCu0RkI/ABZ+e7uBbwuVFQVfmlo84qVcJEJNQYc8p9drAK6GyMOeSeb2CJe/1CN7ZzPuMr4Ckfnn9clTP6NJRSJW+Be0Iaf+BF9xkHxpg0EXkO1zzcey+0s3uipLlaKFRZomcWSimlCqT3LJRSShVIi4VSSqkCabFQSilVIC0WSimlCqTFQimlVIG0WCillCrQ/wf5WgUUDhBLqgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot CV误差曲线\n",
    "test_means = grid.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid.cv_results_[ 'std_test_score' ]\n",
    "train_means = grid.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = grid.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "\n",
    "# plot results\n",
    "n_Cs = len(Cs)\n",
    "number_penaltys = len(penaltys)\n",
    "test_scores = np.array(test_means).reshape(n_Cs,number_penaltys)\n",
    "train_scores = np.array(train_means).reshape(n_Cs,number_penaltys)\n",
    "test_stds = np.array(test_stds).reshape(n_Cs,number_penaltys)\n",
    "train_stds = np.array(train_stds).reshape(n_Cs,number_penaltys)\n",
    "\n",
    "x_axis = np.log10(Cs)\n",
    "for i, value in enumerate(penaltys):\n",
    "    #pyplot.plot(log(Cs), test_scores[i], label= 'penalty:'   + str(value))\n",
    "    plt.errorbar(x_axis, -test_scores[:,i], yerr=test_stds[:,i] ,label = penaltys[i] +' Test')\n",
    "    plt.errorbar(x_axis, -train_scores[:,i], yerr=train_stds[:,i] ,label = penaltys[i] +' Train')\n",
    "    \n",
    "plt.legend()\n",
    "plt.xlabel( 'log(C)' )                                                                                                      \n",
    "plt.ylabel( 'neg-logloss' )\n",
    "plt.savefig('LogisticGridSearchCV_C.png' )\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上图给出了L1正则和L2正则下、不同正则参数C对应的模型在训练集、试集上的正确率（score）。可以看出在训练集和测试集上，L2比L1正则的模型都要相对稳定；但在测试集上当C在1附近时性能最好，此时使用的是L1正则项\n",
    "   \n",
    "  \n",
    "如果最佳值在候选参数的边缘，最好再尝试更大的候选参数或更小的候选参数，直到找到拐点。  \n",
    "\n",
    "这里训练集和测试集分别是l1, c=1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 参数调优 - 1 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5, error_score='raise',\n",
       "       estimator=LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False),\n",
       "       fit_params=None, iid=True, n_jobs=1,\n",
       "       param_grid={'penalty': ['l1', 'l2'], 'C': array([1.     , 1.08163, 1.16327, 1.2449 , 1.32653, 1.40816, 1.4898 ,\n",
       "       1.57143, 1.65306, 1.73469, 1.81633, 1.89796, 1.97959, 2.06122,\n",
       "       2.14286, 2.22449, 2.30612, 2.38776, 2.46939, 2.55102, 2.63265,\n",
       "       2.71429, 2.79592, 2.87755, 2.95918, 3.040...31, 4.34694,\n",
       "       4.42857, 4.5102 , 4.59184, 4.67347, 4.7551 , 4.83673, 4.91837,\n",
       "       5.     ])},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
       "       scoring='neg_log_loss', verbose=0)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Cs1 = np.linspace(1, 5)\n",
    "tuned_parameters1 = dict(penalty = penaltys, C = Cs1)\n",
    "\n",
    "grid1= GridSearchCV(lr_penalty, tuned_parameters1,cv=5, scoring='neg_log_loss')\n",
    "grid1.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.4760268946830622\n",
      "{'C': 1.0, 'penalty': 'l1'}\n"
     ]
    }
   ],
   "source": [
    "print(-grid1.best_score_)\n",
    "print(grid1.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此时最佳参数仍是C=1 和penalty为L1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 参数调优 - 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5, error_score='raise',\n",
       "       estimator=LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False),\n",
       "       fit_params=None, iid=True, n_jobs=1,\n",
       "       param_grid={'penalty': ['l1', 'l2'], 'C': array([0.1    , 0.11837, 0.13673, 0.1551 , 0.17347, 0.19184, 0.2102 ,\n",
       "       0.22857, 0.24694, 0.26531, 0.28367, 0.30204, 0.32041, 0.33878,\n",
       "       0.35714, 0.37551, 0.39388, 0.41224, 0.43061, 0.44898, 0.46735,\n",
       "       0.48571, 0.50408, 0.52245, 0.54082, 0.559...69, 0.85306,\n",
       "       0.87143, 0.8898 , 0.90816, 0.92653, 0.9449 , 0.96327, 0.98163,\n",
       "       1.     ])},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
       "       scoring='neg_log_loss', verbose=0)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Cs2 = np.linspace(0.1, 1)\n",
    "tuned_parameters2 = dict(penalty = penaltys, C = Cs2)\n",
    "\n",
    "grid2= GridSearchCV(lr_penalty, tuned_parameters2,cv=5, scoring='neg_log_loss')\n",
    "grid2.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.47552231265342876\n",
      "{'C': 0.3204081632653062, 'penalty': 'l1'}\n"
     ]
    }
   ],
   "source": [
    "print(-grid2.best_score_)\n",
    "print(grid2.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 可以看出，随着参数C此时最佳参数C=0.32 和penalty为L1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 正则化的 Logistic Regression\n",
    "### L1正则"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegressionCV(Cs=[0.001, 0.01, 0.1, 1, 10, 100, 1000],\n",
       "           class_weight=None, cv=5, dual=False, fit_intercept=True,\n",
       "           intercept_scaling=1.0, max_iter=100, multi_class='ovr',\n",
       "           n_jobs=1, penalty='l1', random_state=None, refit=True,\n",
       "           scoring='neg_log_loss', solver='liblinear', tol=0.0001,\n",
       "           verbose=0)"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegressionCV\n",
    "\n",
    "Cs = [0.001, 0.01, 0.1, 1, 10, 100, 1000]\n",
    "\n",
    "# 大量样本、高维度，L1正则 --> 可选用saga优化求解器\n",
    "# LogisticRegressionCV比GridSearchCV快\n",
    "lrcv_L1 = LogisticRegressionCV(Cs=Cs, cv = 5, scoring='neg_log_loss', penalty='l1', solver='liblinear', multi_class='ovr')\n",
    "lrcv_L1.fit(X_train, y_train)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{1: array([[-0.69314718, -0.6359066 , -0.48922491, -0.48815901, -0.48903195,\n",
       "         -0.48914792, -0.4891567 ],\n",
       "        [-0.69314718, -0.64460736, -0.52487968, -0.52916871, -0.53103365,\n",
       "         -0.53123399, -0.53125234],\n",
       "        [-0.69314718, -0.63328109, -0.45825181, -0.45561724, -0.45589172,\n",
       "         -0.4559245 , -0.45592905],\n",
       "        [-0.69314718, -0.63024185, -0.43336123, -0.4223741 , -0.42195653,\n",
       "         -0.42192426, -0.42191961],\n",
       "        [-0.69314718, -0.63580105, -0.48686008, -0.48452492, -0.48411397,\n",
       "         -0.4841093 , -0.48411177]])}"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lrcv_L1.scores_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAE5lJREFUeJzt3X+wZ/Vd3/HniyVLFIJN5TYSWLMkWU1MQ+JwS8YuMnYqdus0i4oNNNBmowxN7Q5arUpqWgZSp3ViGB2lHSCuEscElARdQpSUSIK2xex30xVYVuJmU7oLJLmBYmBCwYV3//iejd9c7t7Pd+/ew/d77z4fM3f4fs75nPN9n1n2vvZzfnxOqgpJkhZz3KQLkCRNP8NCktRkWEiSmgwLSVKTYSFJajIsJElNhoUkqcmwkCQ1GRaSpKbjJ13AcjnllFNq/fr1ky5DklaUnTt3fqWqZlr9Vk1YrF+/nsFgMOkyJGlFSfLQOP08DSVJajIsJElNhoUkqcmwkCQ1GRaSpCbDQpLUZFhIkpoMC0lSk2EhSWoyLCRJTYaFJKnJsJAkNRkWkqQmw0KS1GRYSJKaDAtJUpNhIUlqMiwkSU2GhSSpybCQJDUZFpKkJsNCktRkWEiSmgwLSVKTYSFJajIsJElNhoUkqcmwkCQ1GRaSpCbDQpLU1GtYJNmU5MEke5NcscD6LUnmkuzqfi4dWffcyPLtfdYpSVrc8X3tOMka4FrgPOAAsCPJ9qp6YF7Xm6tq6wK7eLqq3txXfZKk8fU5sjgb2FtV+6rqWeAm4Pwev0+S1JM+w+I0YP9I+0C3bL4Lktyb5JYk60aWvzTJIMk9SX6oxzolSQ19hkUWWFbz2rcB66vqTOBO4MaRdd9eVbPA24FfSfKaF3xBclkXKIO5ubnlqluSNE+fYXEAGB0pnA48Mtqhqh6rqme65g3AWSPrHun+uw/4FPDd87+gqq6vqtmqmp2ZmVne6iVJX9dnWOwANiQ5I8la4CLgG+5qSnLqSHMzsKdb/vIkJ3SfTwE2AvMvjEuSXiS93Q1VVQeTbAXuANYA26pqd5KrgUFVbQcuT7IZOAg8DmzpNn89cF2S5xkG2n9e4C4qSdKLJFXzLyOsTLOzszUYDCZdhiStKEl2dteHF+UT3JKkJsNCktRkWEiSmgwLSVKTYSFJajIsJElNhoUkqcmwkCQ1GRaSpCbDQpLUZFhIkpoMC0lSk2EhSWoyLCRJTYaFJKnJsJAkNRkWkqQmw0KS1GRYSJKaDAtJUpNhIUlqMiwkSU2GhSSpybCQJDUZFpKkJsNCktRkWEiSmgwLSVKTYSFJauo1LJJsSvJgkr1Jrlhg/ZYkc0l2dT+Xzlt/cpKHk/x6n3VKkhZ3fF87TrIGuBY4DzgA7EiyvaoemNf15qraepjdvBf4dF81SpLG0+fI4mxgb1Xtq6pngZuA88fdOMlZwCuAT/RUnyRpTH2GxWnA/pH2gW7ZfBckuTfJLUnWASQ5Dng/8LOLfUGSy5IMkgzm5uaWq25J0jx9hkUWWFbz2rcB66vqTOBO4MZu+U8AH6+q/Syiqq6vqtmqmp2ZmTnqgiVJC+vtmgXDkcS6kfbpwCOjHarqsZHmDcAvdZ+/B/jeJD8BnASsTfJUVb3gIrkkqX99hsUOYEOSM4CHgYuAt492SHJqVT3aNTcDewCq6uKRPluAWYNCkiant7CoqoNJtgJ3AGuAbVW1O8nVwKCqtgOXJ9kMHAQeB7b0VY8kaelSNf8ywso0Oztbg8Fg0mVI0oqSZGdVzbb6+QS3JKnJsJAkNRkWkqQmw0KS1DRWWCTZmOTE7vMlSa5J8qp+S5MkTYtxRxb/FfhakjcBPwc8BHywt6okSVNl3LA4WMN7bM8HfrWqfhV4WX9lSZKmybgP5T2Z5N3AJcC53fTjL+mvLEnSNBl3ZHEh8Azw41X1RYazx76vt6okSVNl7JEFw9NPzyX5DuB1wIf7K0uSNE3GHVncDZyQ5DTgk8A7gd/qqyhJ0nQZNyxSVV8DfgT4tar6YeAN/ZUlSZomY4dFku8BLgZu75at6ackSdK0GTcsfgp4N3BrN834q4G7+itLkjRNxrrAXVWfBj6d5GVJTqqqfcDl/ZYmSZoW40738cYk/wu4H3ggyc4kXrOQpGPEuKehrgN+uqpeVVXfDvwMw3dmS5KOAeOGxYlV9fVrFFX1KeDEXiqSJE2dcR/K25fk3wO/3bUvAb7QT0mSpGkz7sjix4AZ4KPArd3nd/ZVlCRpuox7N9T/xbufJOmYtWhYJLkNqMOtr6rNy16RJGnqtEYWv/yiVCFJmmqLhkX3MJ4k6Rg31jWLJPfxwtNRfwUMgP9YVY8td2GSpOkx7q2zfwg8B3yoa18EhGFg/Bbw1mWvTJI0NcYNi41VtXGkfV+S/15VG5Nc0kdhkqTpMe5zFiclecuhRpKzgZO65sFlr0qSNFXGDYtLgQ8k+UKS/w18ALg0yYnAfzrcRkk2JXkwyd4kVyywfkuSuSS7up9Lu+Wv6iYr3JVkd5J3HfmhSZKWy7gP5e0A3pjkWxi+Ne+JkdW/u9A2SdYA1wLnAQeAHUm2V9UD87reXFVb5y17FPj7VfVMkpOA+7ttHxmn3iN11W27eeCRr/axa0nq3Xe98mSufGu/E4GPO0X5tyS5huH7t+9M8v4uOBZzNrC3qvZV1bPATcD543xfVT1bVc90zRPGrVOS1I9xL3BvY/gui7d17X8O/CbDd3IfzmnA/pH2AeAtC/S7IMm5wOeAf1NV+wGSrGP4CtfXAj/b16gC6D2RJWmlG/df7K+pqiu7UcK+qroKeHVjmyywbP6zGrcB66vqTOBO4Mavd6za3y1/LfCOJK94wRcklyUZJBnMzc2NeSiSpCM1blg8neScQ40kG4GnG9scANaNtE8HvmF0UFWPjZxuugE4a/5OuhHFbuB7F1h3fVXNVtXszMzMWAciSTpy456G+lfAjYcucAOPA1sa2+wANiQ5A3iY4YN8bx/tkOTUqnq0a24G9nTLTwceq6qnk7wc2AhcM2atkqRlNu7dULuANyU5uWs3bx2qqoNJtgJ3AGuAbVW1O8nVwKCqtgOXJ9nM8FmN0QB6PfD+JMUwnH65qu47skOTJC2XVB12BnKS/PRiG1fV1Pxrf3Z2tgaDwaTLkKQVJcnOqppt9WuNLF62TPVIklaw1hTlV71YhUiSptcRP+yW5LN9FCJJml5LeTJ6oecnJEmr2FLC4vZlr0KSNNWOOCyq6j19FCJJml7jvlb1SQ7/WtWfqap9y12YJGl6jPsE9zUMp+r4EMNrFhcB3wY8yHCSwe/rozhJ0nQY9zTUpqq6rqqerKqvVtX1wA9W1c3Ay3usT5I0BcYNi+eTvC3Jcd3P20bWHf4RcEnSqjBuWFzM8B0WXwa+1H2+JMk3AfPfcidJWmXGnUhwH/DWw6z+0+UrR5I0jcZ9rep3JPlkkvu79plJvIVWko4R456GugF4N/DXAFV1L8M7oiRJx4Bxw+Kbq+oz85YdXO5iJEnTadyw+EqS19Dd+ZTkR4FHF99EkrRajPtQ3r8Grgdel+Rh4AsM75CSJB0Dxg2Lh4HfBO4C/jbwVeAdwNU91SVJmiLjhsUfAE8An2U47Yck6RgyblicXlWbeq1EkjS1xr3A/T+SvLHXSiRJU2vckcU5wJYkXwCeYTjzbFXVmb1VJkmaGuOGxT/utQpJ0lQbd26oh/ouRJI0vZbyDm5J0jHGsJAkNRkWkqQmw0KS1NRrWCTZlOTBJHuTXLHA+i1J5pLs6n4u7Za/Ocn/TLI7yb1JLuyzTknS4sa9dfaIJVkDXAucBxwAdiTZXlUPzOt6c1XNfzXr14B/UVV/meSVwM4kd1TVE33VK0k6vD5HFmcDe6tqX1U9C9wEnD/OhlX1uar6y+7zIwzf/T3TW6WSpEX1GRanAftH2ge6ZfNd0J1quiXJuvkrk5wNrAU+30+ZkqSWPsMiCyyree3bgPXdtCF3Ajd+ww6SU4HfBt5ZVc+/4AuSy5IMkgzm5uaWqWxJ0nx9hsUBYHSkcDrzpjevqseq6pmueQNw1qF1SU4GbgfeU1X3LPQFVXV9Vc1W1ezMjGepJKkvfYbFDmBDkjOSrAUuAraPduhGDodsBvZ0y9cCtwIfrKrf67FGSdIYersbqqoOJtkK3AGsAbZV1e4kVwODqtoOXJ5kM3AQeBzY0m3+NuBc4FuTHFq2pap29VWvJOnwUjX/MsLKNDs7W4PBYNJlSNKKkmRnVc22+vkEtySpybCQJDUZFpKkJsNCktRkWEiSmgwLSVKTYSFJajIsJElNhoUkqcmwkCQ1GRaSpCbDQpLUZFhIkpoMC0lSk2EhSWoyLCRJTYaFJKnJsJAkNRkWkqQmw0KS1GRYSJKaDAtJUpNhIUlqMiwkSU2GhSSpybCQJDUZFpKkJsNCktRkWEiSmnoNiySbkjyYZG+SKxZYvyXJXJJd3c+lI+v+KMkTST7WZ42SpLbj+9pxkjXAtcB5wAFgR5LtVfXAvK43V9XWBXbxPuCbgX/ZV42SpPH0ObI4G9hbVfuq6lngJuD8cTeuqk8CT/ZVnCRpfH2GxWnA/pH2gW7ZfBckuTfJLUnWHckXJLksySDJYG5u7mhqlSQtos+wyALLal77NmB9VZ0J3AnceCRfUFXXV9VsVc3OzMwssUxJUkufYXEAGB0pnA48Mtqhqh6rqme65g3AWT3WI0laoj7DYgewIckZSdYCFwHbRzskOXWkuRnY02M9kqQl6u1uqKo6mGQrcAewBthWVbuTXA0Mqmo7cHmSzcBB4HFgy6Htk/wJ8DrgpCQHgB+vqjv6qleSdHipmn8ZYWWanZ2twWAw6TIkaUVJsrOqZlv9fIJbktRkWEiSmgwLSVKTYSFJajIsJElNhoUkqcmwkCQ1GRaSpCbDQpLUZFhIkpoMC0lSk2EhSWoyLCRJTYaFJKnJsJAkNRkWkqQmw0KS1GRYSJKaDAtJUpNhIUlqMiwkSU2GhSSpybCQJDUZFpKkJsNCktRkWEiSmgwLSVJTqmrSNSyLJHPAQ0exi1OAryxTOZO0Wo4DPJZptVqOZbUcBxzdsbyqqmZanVZNWBytJIOqmp10HUdrtRwHeCzTarUcy2o5DnhxjsXTUJKkJsNCktRkWPyN6yddwDJZLccBHsu0Wi3HslqOA16EY/GahSSpyZGFJKnJsOgkeW+Se5PsSvKJJK+cdE1LleR9Sf6iO55bk/ytSde0VEn+aZLdSZ5PsuLuXEmyKcmDSfYmuWLS9RyNJNuSfDnJ/ZOu5WgkWZfkriR7uv+3fnLSNS1Vkpcm+UySP++O5arevsvTUENJTq6qr3afLwe+q6reNeGyliTJDwB/XFUHk/wSQFX9/ITLWpIkrweeB64D/m1VDSZc0tiSrAE+B5wHHAB2AP+sqh6YaGFLlORc4Cngg1X1dyddz1IlORU4tao+m+RlwE7gh1bin0uSACdW1VNJXgL8KfCTVXXPcn+XI4vOoaDonAis2BStqk9U1cGueQ9w+iTrORpVtaeqHpx0HUt0NrC3qvZV1bPATcD5E65pyarqbuDxSddxtKrq0ar6bPf5SWAPcNpkq1qaGnqqa76k++nld5dhMSLJLybZD1wM/IdJ17NMfgz4w0kXcYw6Ddg/0j7ACv2ltFolWQ98N/Bnk61k6ZKsSbIL+DLw36qql2M5psIiyZ1J7l/g53yAqvqFqloH/A6wdbLVLq51LF2fXwAOMjyeqTXOsaxQWWDZih2xrjZJTgI+AvzUvDMLK0pVPVdVb2Z4BuHsJL2cIjy+j51Oq6r6/jG7fgi4Hbiyx3KOSutYkrwD+CfAP6wpvzB1BH8uK80BYN1I+3TgkQnVohHd+f2PAL9TVR+ddD3LoaqeSPIpYBOw7DchHFMji8Uk2TDS3Az8xaRqOVpJNgE/D2yuqq9Nup5j2A5gQ5IzkqwFLgK2T7imY153Ufg3gD1Vdc2k6zkaSWYO3e2Y5JuA76en313eDdVJ8hHgOxneefMQ8K6qeniyVS1Nkr3ACcBj3aJ7VvCdXT8M/BowAzwB7KqqfzTZqsaX5AeBXwHWANuq6hcnXNKSJfkw8H0MZzj9EnBlVf3GRItagiTnAH8C3Mfw7zvAv6uqj0+uqqVJciZwI8P/v44Dfreqru7luwwLSVKLp6EkSU2GhSSpybCQJDUZFpKkJsNCktRkWEhHIMlT7V6Lbn9Lkld3n09Kcl2Sz3czht6d5C1J1nafj6mHZjXdDAvpRZLkDcCaqtrXLfoAw4n5NlTVG4AtwCndpIOfBC6cSKHSAgwLaQky9L5uDqv7klzYLT8uyX/pRgofS/LxJD/abXYx8Addv9cAbwHeU1XPA3Sz097e9f39rr80FRzmSkvzI8CbgTcxfKJ5R5K7gY3AeuCNwN9hOP31tm6bjcCHu89vYPg0+nOH2f/9wN/rpXJpCRxZSEtzDvDhbsbPLwGfZvjL/Rzg96rq+ar6InDXyDanAnPj7LwLkWe7l/NIE2dYSEuz0PTjiy0HeBp4afd5N/CmJIv9HTwB+H9LqE1adoaFtDR3Axd2L56ZAc4FPsPwtZYXdNcuXsFw4r1D9gCvBaiqzwMD4KpuFlSSbDj0Do8k3wrMVdVfv1gHJC3GsJCW5lbgXuDPgT8Gfq477fQRhu+xuJ/he8P/DPirbpvb+cbwuBT4NmBvkvuAG/ib9138A2DFzYKq1ctZZ6VlluSkqnqqGx18BthYVV/s3jdwV9c+3IXtQ/v4KPDuFfz+ca0y3g0lLb+PdS+kWQu8txtxUFVPJ7mS4Xu4/8/hNu5elPT7BoWmiSMLSVKT1ywkSU2GhSSpybCQJDUZFpKkJsNCktRkWEiSmv4/vWGMNuugrs8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "C is: [1.]\n"
     ]
    }
   ],
   "source": [
    "n_Cs = len(Cs)\n",
    "n_classes = 5\n",
    "scores =  np.zeros((n_classes,n_Cs))\n",
    "\n",
    "for j in range(n_classes):\n",
    "        scores[j][:] = np.mean(lrcv_L1.scores_[1][j],axis = 0)\n",
    "    \n",
    "mse_mean = -np.mean(scores, axis = 0)\n",
    "plt.plot(np.log10(Cs), mse_mean.reshape(n_Cs,1)) \n",
    "#plt.plot(np.log10(reg.Cs)*np.ones(3), [0.28, 0.29, 0.30])\n",
    "plt.xlabel('log(C)')\n",
    "plt.ylabel('neg-logloss')\n",
    "plt.show()\n",
    "\n",
    "print ('C is:',lrcv_L1.C_)  #对多类分类问题，每个类别的分类器有一个C"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.40964937,  1.12849315, -0.09297813,  0.0247598 , -0.08407628,\n",
       "         0.62917109,  0.27932924,  0.14387031]])"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lrcv_L1.coef_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.001 0.5304209374804264\n"
     ]
    }
   ],
   "source": [
    "best_C = np.argmin(mse_mean)\n",
    "best_score = np.min(mse_mean)\n",
    "print (Cs[best_C], best_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用L1正则时，最佳参数C=0.001"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.4.2 L2正则"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegressionCV(Cs=[0.001, 0.01, 0.1, 1, 10, 100, 1000],\n",
       "           class_weight=None, cv=5, dual=False, fit_intercept=True,\n",
       "           intercept_scaling=1.0, max_iter=100, multi_class='ovr',\n",
       "           n_jobs=1, penalty='l2', random_state=None, refit=True,\n",
       "           scoring='neg_log_loss', solver='liblinear', tol=0.0001,\n",
       "           verbose=0)"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegressionCV\n",
    "\n",
    "Cs = [0.001, 0.01, 0.1, 1, 10, 100, 1000]\n",
    "\n",
    "# 大量样本（6W+）、高维度（93），L2正则 --> 缺省用lbfgs，为了和GridSeachCV比较，也用liblinear\n",
    "\n",
    "lr_cv_L2 = LogisticRegressionCV(Cs=Cs, cv = 5, scoring='neg_log_loss', penalty='l2', solver='liblinear', multi_class='ovr')\n",
    "lr_cv_L2.fit(X_train, y_train)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{1: array([[-0.62523328, -0.51207677, -0.48350008, -0.48797856, -0.48902949,\n",
       "         -0.4891444 , -0.489156  ],\n",
       "        [-0.6338686 , -0.54322757, -0.52514943, -0.53011593, -0.53113632,\n",
       "         -0.53124692, -0.53125807],\n",
       "        [-0.62566121, -0.50673804, -0.46048749, -0.4562292 , -0.45595951,\n",
       "         -0.45593526, -0.45593286],\n",
       "        [-0.62087342, -0.48731256, -0.42929827, -0.422546  , -0.42197953,\n",
       "         -0.42192491, -0.42191947],\n",
       "        [-0.63268716, -0.52538884, -0.4853734 , -0.48392885, -0.48408647,\n",
       "         -0.48410727, -0.48410941]])}"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_cv_L2.scores_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAFQVJREFUeJzt3X+w3XV95/Hni2CoAra2xIoQDdq0FQV1uI27C+u4bdG0sw3bYoWudIktS93ZDO3WnRa2bhlgOzutlq7ToTtCGxc7raAobQAtFYrSrYvmxuVXiNQYS7mi5Qrl16rEwHv/ON+kx8vN/Zzc3G/OvcnzMXNmzufz/XzOeX8HuC8+3+/3fL+pKiRJmsth4y5AkrT4GRaSpCbDQpLUZFhIkpoMC0lSk2EhSWoyLCRJTYaFJKnJsJAkNR0+7gIWyjHHHFOrVq0adxmStKRs2bLl61W1ojXuoAmLVatWMTk5Oe4yJGlJSfLAKON6PQyVZG2S+5NsT3LhLNvXJ5lOcmf3Oq/rf3mSLV3f1iTv7LNOSdLceltZJFkGXAGcDkwBm5Nsqqr7Zgy9tqo2zOj7KvAvqurpJEcB93ZzH+qrXknS3vW5slgDbK+qHVW1E7gGOGOUiVW1s6qe7ppH4Il4SRqrPv8IHwc8ONSe6vpmOjPJ3UmuS7Jyd2eSlUnu7j7jt2dbVSQ5P8lkksnp6emFrl+S1OkzLDJL38yHZ9wArKqqk4FbgKv3DKx6sOv/AeDcJN//nA+rurKqJqpqYsWK5sl8SdI89RkWU8DKofbxwHesDqrqkaHDTVcBp8z8kG5FsRX4lz3VKUlq6DMsNgOrk5yQZDlwNrBpeECSY4ea64BtXf/xSZ7fvX8RcCpwf4+1SpLm0NvVUFW1K8kG4GZgGbCxqrYmuRSYrKpNwAVJ1gG7gEeB9d30VwG/m6QYHM56b1Xd01etkqS55WB5BvfExET5ozxJ2jdJtlTVRGucl6RKkpoMC0lSk2EhSWoyLCRJTYaFJKnJsJAkNRkWkqQmw0KS1GRYSJKaDAtJUpNhIUlqMiwkSU2GhSSpybCQJDUZFpKkJsNCktRkWEiSmgwLSVKTYSFJajIsJElNhoUkqcmwkCQ1GRaSpCbDQpLUZFhIkpoMC0lSk2EhSWoyLCRJTYaFJKnJsJAkNfUaFknWJrk/yfYkF86yfX2S6SR3dq/zuv7XJfk/SbYmuTvJWX3WKUma2+F9fXCSZcAVwOnAFLA5yaaqum/G0GurasOMvm8A/66qvpjkpcCWJDdX1WN91StJ2rs+VxZrgO1VtaOqdgLXAGeMMrGq/raqvti9fwh4GFjRW6WSpDn1GRbHAQ8Otae6vpnO7A41XZdk5cyNSdYAy4Ev9VOmJKmlz7DILH01o30DsKqqTgZuAa7+jg9IjgX+GHhHVT37nC9Izk8ymWRyenp6gcqWJM3UZ1hMAcMrheOBh4YHVNUjVfV017wKOGX3tiQvBG4C3l1Vd8z2BVV1ZVVNVNXEihUepZKkvvQZFpuB1UlOSLIcOBvYNDygWznstg7Y1vUvB64HPlhVH+mxRknSCHq7GqqqdiXZANwMLAM2VtXWJJcCk1W1CbggyTpgF/AosL6b/jbgjcD3Jdndt76q7uyrXknS3qVq5mmEpWliYqImJyfHXYYkLSlJtlTVRGucv+CWJDUZFpKkJsNCktRkWEiSmgwLSVKTYSFJajIsJElNhoUkqcmwkCQ1GRaSpCbDQpLUZFhIkpoMC0lSk2EhSWoyLCRJTYaFJKnJsJAkNRkWkqQmw0KS1GRYSJKaDAtJUpNhIUlqMiwkSU2GhSSpybCQJDUZFpKkJsNCktRkWEiSmgwLSVKTYSFJauo1LJKsTXJ/ku1JLpxl+/ok00nu7F7nDW37iySPJbmxzxolSW2H9/XBSZYBVwCnA1PA5iSbquq+GUOvraoNs3zEe4AXAL/UV42SpNH0ubJYA2yvqh1VtRO4Bjhj1MlVdSvwZF/FSZJGN1JYJDk1yZHd+3OSXJ7k5Y1pxwEPDrWnur6Zzkxyd5LrkqwcqWpJ0gE16srifwLfSPJa4NeAB4APNuZklr6a0b4BWFVVJwO3AFePWM/gC5Lzk0wmmZyent6XqZKkfTBqWOyqqmJwGOl9VfU+4OjGnClgeKVwPPDQ8ICqeqSqnu6aVwGnjFjP7vlXVtVEVU2sWLFiX6ZKkvbBqGHxZJKLgHOAm7qT189rzNkMrE5yQpLlwNnApuEBSY4daq4Dto1YjyTpABo1LM4CngZ+saq+xuDcw3vmmlBVu4ANwM0MQuDDVbU1yaVJ1nXDLkiyNcldwAXA+t3zk/w18BHgx5JMJXnLPuyXJGkBZXB0qTFocHL7W1X1TJIfBH4Y+ERVfbvvAkc1MTFRk5OT4y5DkpaUJFuqaqI1btSVxe3AEUmOA24F3gH8r/mXJ0laSkYNi1TVN4CfAX6/qn4aeHV/ZUmSFpORwyLJPwfeDtzU9S3rpyRJ0mIzalj8CnARcH13kvoVwG39lSVJWkxGujdUVX0a+HSSo5McVVU7GFy9JEk6BIx6u4+Tkvxf4F7gviRbknjOQpIOEaMehno/8KtV9fKqehnwLga/uJYkHQJGDYsjq2rPOYqq+hRwZC8VSZIWnVGfZ7EjyX8F/rhrnwN8uZ+SJEmLzagri18AVgAfA67v3r+jr6IkSYvLqFdD/SNe/SRJh6w5wyLJDTz3GRR7VNW6vW2TJB08WiuL9x6QKiRJi9qcYdH9GE+SdIgb6ZxFknt47uGox4FJ4L9V1SMLXZgkafEY9dLZTwDPAH/atc9m8IztxxncqvynFrwySdKiMWpYnFpVpw6170nyN1V1apJz+ihMkrR4jPo7i6OSvGF3I8ka4KiuuWvBq5IkLSqjrizOAzYmOYrB4acngF/sHrf63/sqTpK0OIz6o7zNwElJvpvBU/MeG9r84V4qO4AuuWEr9z30xLjLkKR5OfGlL+Tin+r3RuCj3qL8u5NczuD527ck+d0uOCRJh4BRD0NtZPAsi7d17Z8HPsDgmdxLXt+JLElL3ahh8cqqOnOofUmSO/soSJK0+Ix6NdQ3k5y2u5HkVOCb/ZQkSVpsRl1Z/Afg6t0nuIFHgfV9FSVJWlxGvRrqTuC1SV7Ytb10SJIOIa1blP/qXvoBqKrLe6hJkrTItFYWRx+QKiRJi1rrFuWXHKhCJEmL16hXQ+2R5PN9FCJJWrz2OSwYXA012sBkbZL7k2xPcuEs29cnmU5yZ/c6b2jbuUm+2L3OnUedkqQFMuqls8NuGmVQkmXAFcDpwBSwOcmmqrpvxtBrq2rDjLnfC1wMTDB46NKWbu4/zqNeSdJ+2ueVRVW9e8Sha4DtVbWjqnYC1wBnjDj3LcAnq+rRLiA+Cazd11olSQtj1BsJPpnkiRmvB5Ncn+QVe5l2HPDgUHuq65vpzCR3J7kuycp9nCtJOgBGPQx1OfAQg8eqhsFjVV8C3M/gJoNvmmXObOc2Zj7H+wbgQ1X1dJJ3AlcDPzriXJKcD5wP8LKXvWyU/ZAkzcOoh6HWVtX7q+rJqnqiqq4EfrKqrgVetJc5U8DKofbxDAJnj6p6pKqe7ppXAaeMOrebf2VVTVTVxIoVK0bcFUnSvho1LJ5N8rYkh3Wvtw1te87/8Xc2A6uTnJBkOYPVyKbhAUmOHWquA7Z1728G3pzkRUleBLy565MkjcGoh6HeDrwP+AMG4XAHcE6S5wMbZptQVbuSbGDwR34ZsLGqtia5FJisqk3ABUnWMXiO956bE1bVo0kuYxA4AJdW1aPz2UFJ0v5L1d4WBkvLxMRETU5OjrsMSVpSkmypqonWuFGvhvrBJLcmubdrn5xk1EtoJUlL3KjnLK4CLgK+DVBVdzM4ByFJOgSMGhYvqKrPzejbtdDFSJIWp1HD4utJXkl35VOStwJf7a0qSdKiMurVUP8RuBL44SRfAb7M4AopSdIhYNSw+ArwAeA24HuBJ4BzgUt7qkuStIiMGhZ/DjwGfJ5ZfkktSTq4jRoWx1eVd32VpEPUqCe4P5PkpF4rkSQtWqOuLE4D1if5MvA0g7vCVlWd3FtlkqRFY9Sw+Ileq5AkLWojhUVVPdB3IZKkxWufH6sqSTr0GBaSpCbDQpLUZFhIkpoMC0lSk2EhSWoyLCRJTYaFJKnJsJAkNRkWkqQmw0KS1GRYSJKaDAtJUpNhIUlqMiwkSU2GhSSpybCQJDUZFpKkJsNCktTUa1gkWZvk/iTbk1w4x7i3JqkkE117eZIPJLknyV1J3tRnnZKkuR3e1wcnWQZcAZwOTAGbk2yqqvtmjDsauAD47FD3vweoqpOSvBj4RJIfqapn+6pXkrR3fa4s1gDbq2pHVe0ErgHOmGXcZcDvAN8a6jsRuBWgqh4GHgMmeqxVkjSHPsPiOODBofZU17dHktcDK6vqxhlz7wLOSHJ4khOAU4CVM78gyflJJpNMTk9PL2z1kqQ9ejsMBWSWvtqzMTkM+D1g/SzjNgKvAiaBB4DPALue82FVVwJXAkxMTNTM7ZKkhdFnWEzxnauB44GHhtpHA68BPpUE4CXApiTrqmoS+E+7Byb5DPDFHmuVJM2hz8NQm4HVSU5Ishw4G9i0e2NVPV5Vx1TVqqpaBdwBrKuqySQvSHIkQJLTgV0zT4xLkg6c3lYWVbUryQbgZmAZsLGqtia5FJisqk1zTH8xcHOSZ4GvAD/fV52SpLY+D0NRVR8HPj6j7zf3MvZNQ+//DvihPmuTJI3OX3BLkpoMC0lSk2EhSWoyLCRJTYaFJKnJsJAkNRkWkqQmw0KS1GRYSJKaDAtJUpNhIUlqMiwkSU2GhSSpybCQJDUZFpKkJsNCktRkWEiSmgwLSVKTYSFJajIsJElNhoUkqcmwkCQ1GRaSpCbDQpLUZFhIkpoMC0lSk2EhSWoyLCRJTYaFJKnJsJAkNfUaFknWJrk/yfYkF84x7q1JKslE135ekquT3JNkW5KL+qxTkjS33sIiyTLgCuAngBOBn0ty4izjjgYuAD471P2zwBFVdRJwCvBLSVb1VaskaW59rizWANurakdV7QSuAc6YZdxlwO8A3xrqK+DIJIcDzwd2Ak/0WKskaQ59hsVxwIND7amub48krwdWVtWNM+ZeB/w/4KvA3wPvrapHe6xVkjSHPsMis/TVno3JYcDvAe+aZdwa4BngpcAJwLuSvOI5X5Ccn2QyyeT09PTCVC1Jeo4+w2IKWDnUPh54aKh9NPAa4FNJ/g74Z8Cm7iT3vwX+oqq+XVUPA38DTMz8gqq6sqomqmpixYoVPe2GJKnPsNgMrE5yQpLlwNnApt0bq+rxqjqmqlZV1SrgDmBdVU0yOPT0oxk4kkGQfKHHWiVJc+gtLKpqF7ABuBnYBny4qrYmuTTJusb0K4CjgHsZhM4HquruvmqVJM0tVdUetQRMTEzU5OTkuMuQpCUlyZaqes5h/pn8BbckqcmwkCQ1GRaSpCbDQpLUZFhIkpoMC0lSk2EhSWoyLCRJTYaFJKnJsJAkNRkWkqSmg+beUEmmgQf24yOOAb6+QOWM08GyH+C+LFYHy74cLPsB+7cvL6+q5jMeDpqw2F9JJke5mdZid7DsB7gvi9XBsi8Hy37AgdkXD0NJkpoMC0lSk2HxT64cdwEL5GDZD3BfFquDZV8Olv2AA7AvnrOQJDW5spAkNRkWnSSXJbk7yZ1J/jLJS8dd03wleU+SL3T7c32S7xl3TfOV5GeTbE3ybJIld+VKkrVJ7k+yPcmF465nfyTZmOThJPeOu5b9kWRlktuSbOv+3frlcdc0X0m+K8nnktzV7cslvX2Xh6EGkrywqp7o3l8AnFhV7xxzWfOS5M3AX1XVriS/DVBVvz7msuYlyauAZ4H3A/+5qpbMg9aTLAP+FjgdmAI2Az9XVfeNtbB5SvJG4Cngg1X1mnHXM19JjgWOrarPJzka2AL8m6X4zyVJgCOr6qkkzwP+N/DLVXXHQn+XK4vO7qDoHAks2RStqr+sql1d8w7g+HHWsz+qaltV3T/uOuZpDbC9qnZU1U7gGuCMMdc0b1V1O/DouOvYX1X11ar6fPf+SWAbcNx4q5qfGniqaz6ve/Xyt8uwGJLkt5I8CLwd+M1x17NAfgH4xLiLOEQdBzw41J5iif5ROlglWQW8HvjseCuZvyTLktwJPAx8sqp62ZdDKiyS3JLk3lleZwBU1W9U1UrgT4AN4612bq196cb8BrCLwf4sWqPsyxKVWfqW7Ir1YJPkKOCjwK/MOLKwpFTVM1X1OgZHENYk6eUQ4eF9fOhiVVU/PuLQPwVuAi7usZz90tqXJOcC/xr4sVrkJ6b24Z/LUjMFrBxqHw88NKZaNKQ7vv9R4E+q6mPjrmchVNVjST4FrAUW/CKEQ2plMZckq4ea64AvjKuW/ZVkLfDrwLqq+sa46zmEbQZWJzkhyXLgbGDTmGs65HUnhf8I2FZVl4+7nv2RZMXuqx2TPB/4cXr62+XVUJ0kHwV+iMGVNw8A76yqr4y3qvlJsh04Anik67pjCV/Z9dPA7wMrgMeAO6vqLeOtanRJfhL4H8AyYGNV/daYS5q3JB8C3sTgDqf/AFxcVX801qLmIclpwF8D9zD47x3gv1TVx8dX1fwkORm4msG/X4cBH66qS3v5LsNCktTiYShJUpNhIUlqMiwkSU2GhSSpybCQJDUZFtI+SPJUe9Sc869L8oru/VFJ3p/kS90dQ29P8oYky7v3h9SPZrW4GRbSAZLk1cCyqtrRdf0hgxvzra6qVwPrgWO6mw7eCpw1lkKlWRgW0jxk4D3dPazuSXJW139Ykj/oVgo3Jvl4krd2094O/Hk37pXAG4B3V9WzAN3daW/qxv5ZN15aFFzmSvPzM8DrgNcy+EXz5iS3A6cCq4CTgBczuP31xm7OqcCHuvevZvBr9Gf28vn3Aj/SS+XSPLiykObnNOBD3R0//wH4NIM/7qcBH6mqZ6vqa8BtQ3OOBaZH+fAuRHZ2D+eRxs6wkOZnttuPz9UP8E3gu7r3W4HXJpnrv8EjgG/NozZpwRkW0vzcDpzVPXhmBfBG4HMMHmt5Znfu4vsZ3Hhvt23ADwBU1ZeASeCS7i6oJFm9+xkeSb4PmK6qbx+oHZLmYlhI83M9cDdwF/BXwK91h50+yuA5FvcyeG74Z4HHuzk38Z3hcR7wEmB7knuAq/in5138K2DJ3QVVBy/vOistsCRHVdVT3ergc8CpVfW17nkDt3XtvZ3Y3v0ZHwMuWsLPH9dBxquhpIV3Y/dAmuXAZd2Kg6r6ZpKLGTyH++/3Nrl7UNKfGRRaTFxZSJKaPGchSWoyLCRJTYaFJKnJsJAkNRkWkqQmw0KS1PT/AZrCAE/VojgBAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "C is: [1.]\n"
     ]
    }
   ],
   "source": [
    "n_Cs = len(Cs)\n",
    "n_classes = 5\n",
    "scores =  np.zeros((n_classes,n_Cs))\n",
    "\n",
    "for j in range(n_classes):\n",
    "        scores[j][:] = np.mean(lr_cv_L2.scores_[1][j],axis = 0)\n",
    "    \n",
    "mse_mean = -np.mean(scores, axis = 0)\n",
    "plt.plot(np.log10(Cs), mse_mean.reshape(n_Cs,1)) \n",
    "#plt.plot(np.log10(reg.Cs)*np.ones(3), [0.28, 0.29, 0.30])\n",
    "plt.xlabel('log(C)')\n",
    "plt.ylabel('neg-logloss')\n",
    "plt.show()\n",
    "\n",
    "print ('C is:',lr_cv_L2.C_)  #对多类分类问题，每个类别的分类器有一个C"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.001 0.5035600163280257\n"
     ]
    }
   ],
   "source": [
    "best_C = np.argmin(mse_mean)\n",
    "best_score = np.min(mse_mean)\n",
    "print (Cs[best_C], best_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用L2正则时，最佳参数C=0.001"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 相比较之下还是用GridSearchCV方法下的logistic回归模型做预测"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_test = data['Target']   \n",
    "X_test = data.drop([\"Target\"], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "bm = grid2.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_pred = bm.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "out_df = pd.DataFrame(test_pred)\n",
    "out_df.to_csv(\"test_pred.csv\", index=False)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
